Recent research work has shown that transductive learning/inference outperforms standard methods that were used before, where people embed features in a high dimensional space and then use the distance metric in that space to classify objects. This has been found in few-shot learning, other metric learning works, etc. But all of these results are experimental. Is there any theoretical/mathematical reason why this is the case? Or is that a field of research for now?
If you want to look at such examples, look at section 1.1 of the paper "Transductive Information Maximization For Few-Shot Learning", here is the URL: https://arxiv.org/pdf/2008.11297.pdf
For the other approach, we can look at Matching nets, prototypical networks (https://arxiv.org/pdf/1703.05175.pdf), etc. Recent papers show that features can be embedded in Hyperspheres and Oblique Manifolds(https://openaccess.thecvf.com/content/ICCV2021/papers/Qi_Transductive_Few-Shot_Classification_on_the_Oblique_Manifold_ICCV_2021_paper.pdf) and there we use distances on manifolds in our classification tasks.
There are experiments showing how one method performs with respect to the other. But is there any theoretical justification related to which method should do better and in what circumstances?