I am working with a u-net, a variation of the more commonly known fully convolutional network for semantic segmentation. For training a u-net, I was given the suggestion that I should use a pre-trained network. But, I'm not quite sure how this makes sense.
My understanding is, in pre-training, we freeze the network up to the final classifying layers from a pre-trained network and then, we initialize random weights for the classifying layers. Then, we take our new dataset and retrain the weights through the whole network. Basically, we don't have random initialized weights for the convolutional blocks.
Now, if I'm using something like the u-net which has a very unique structure, how exactly would I do this pre-training with something like vgg16? The only thing I can think of is to share the feature maps during the upsampling layers but that's about it and that's not really pre-trained weights.