0
$\begingroup$

When the architectures of the teacher and student networks do not just vary by network depths but are completely different, is it logically correct to distill knowledge at feature level (say from middle layer) in such scenario? Thanks in advance

$\endgroup$
0
$\begingroup$

There is no such thing as "correct" or "incorrect". The only question is whether it works well enough or not. The only way to figure that out is to try it and see. (It's not clear to me why you'd distill at the middle layer as opposed to the last or next-to-last layer, but you can do it.)

$\endgroup$
1
  • $\begingroup$ Thanks for your response. I have tried distillation from the intermediate layer and the performance has degraded and hence the query. The goal was to enable the student model to learn the encoded richer representation from the teacher model. However, there are several works in the literature that have achieved higher performances using feature distillation from the middle layer. But their networks vary mostly in depth and not in total structure. $\endgroup$
    – Parandrea
    Jun 29 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.