2
$\begingroup$

I'm new to the CS stack exchange, so a fond hello to you all!

I joined since I have a question I've been curious about. I have recently been running some experiments in transfer learning - specifically I train a network to recognise a certain image, let's say a cat, and then I transfer the input and hidden layers of the network to another dataset and see if the trained weight distribution is a better starting point as opposed to a random distribution through XAVIER etc. etc.

I now wonder, what if the input is different, rather than the output? For example, if I have a dataset of Hotel reviews and build a sentiment analysis type predictor that would predict whether a review is negative or positive, one would have a binary classification problem. If you were then to apply those layers and weights as the starting point to Restaurant reviews, one has changed the input attributes (eg. word stems from a different domain) but the output may still be a binary classification problem for positive and negative classes.

All of the examples of transfer learning I see will change the input and output layers while retaining the hidden layers. If the input changes, but the output does not, is it still Transfer Learning?

Thank you!

$\endgroup$
  • $\begingroup$ Hello to you too, and welcome! $\endgroup$ – D.W. Feb 3 at 0:07
0
$\begingroup$

This type of transfer learning only applies if you have the same features. If you don't, you'll either have to do something else, or find some way to first extract a common set of features from both, or choose a feature set that is the union of the two features. The latter choice would handle the example you gave of hotel reviews vs restaurant reviews.

That said, there is emerging evidence that for deep neural networks, it's not clear that transfer learning (pre-training on a different task) provides any advantage in aaccuracy over training from scratch on the second task. Recent research has shown that if you train long enough on the second task, you get accuracy as good as if you start with weights from a network that was pre-trained on the second task. So, it appears the main advantage of this kind of transfer learning might be to speed up training time, rather than to build better models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.