Several papers(1 (originator), 2, 3) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand).
I haven't been able to get clear in my head what this means the actual final layer and output/labels of the neural network are.
For (plain) softmax the activation function is the softmax function: $$\mathbf{\hat{y}}=\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$$
and the loss (error) function is cross entrypy $$C(\mathbf{\hat{y}},\mathbf{y})=\sum_{k=1}^K-\mathbf{y_k}\times \log{\mathbf{\hat{y}_k}}$$ where y is "one-hot" -- all zeros except a 1 for the index matching the class (this lead to effient implementation, if you know the class indexes).
For Hierachical Softmax: What is the form of the label y, the activation function $\sigma(\mathbf{z})$ and the loss (error) function $C(\mathbf{\hat{y}},\mathbf{y})$
I am starting the suspect that the label is a Binary code for the class, eg a Huffman code, and the activation function is simply sigmoid (or tanh) and the loss is just squared error.
Is that all there is too it?
Or is it infact done with a multilayer network, in some way? (Obviously you can't stack softmax layers as inputs to softmax layers).
Implementations
There are quiet a few implementations around, but I find all of them hard to follow.
A even more different Python (Theano) implementation. This one is not truly Hierarchical soft-max as it only has two layers.
Papers
- Morin, F., & Bengio, Y. (2005, January). Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics (pp. 246-252).
- Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119).
- Wang, Y., Li, Z., Liu, J., He, Z., Huang, Y., & Li, D. (2014). Word Vector Modeling for Sentiment Analysis of Product Reviews. In Natural Language Processing and Chinese Computing (pp. 168-180). Springer Berlin Heidelberg.