What is the activation function, label and loss function for Hierachical Softmax

Question

Several papers(1 (originator), 2, 3) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand).

I haven't been able to get clear in my head what this means the actual final layer and output/labels of the neural network are.

For (plain) softmax the activation function is the softmax function: $$\mathbf{\hat{y}}=\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$$

and the loss (error) function is cross entrypy $$C(\mathbf{\hat{y}},\mathbf{y})=\sum_{k=1}^K-\mathbf{y_k}\times \log{\mathbf{\hat{y}_k}}$$ where y is "one-hot" -- all zeros except a 1 for the index matching the class (this lead to effient implementation, if you know the class indexes).

For Hierachical Softmax: What is the form of the label y, the activation function $\sigma(\mathbf{z})$ and the loss (error) function $C(\mathbf{\hat{y}},\mathbf{y})$

I am starting the suspect that the label is a Binary code for the class, eg a Huffman code, and the activation function is simply sigmoid (or tanh) and the loss is just squared error.

Is that all there is too it?

Or is it infact done with a multilayer network, in some way? (Obviously you can't stack softmax layers as inputs to softmax layers).

Implementations

There are quiet a few implementations around, but I find all of them hard to follow.

• Word2Vec in C, and Gensim in Python.

  • I'm not great at understanding C -- too many clever tricks (like using 1D indexing + offsets on 2D arrays), and the python harks close to the C (it is an enhanced translation).
  • There are two linked articles A, B which go someway towards explaining the C code.

• A very different python (Cython actually)

• A even more different Python (Theano) implementation. This one is not truly Hierarchical soft-max as it only has two layers.

---

Papers

1. Morin, F., & Bengio, Y. (2005, January). Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics (pp. 246-252).
2. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119).
3. Wang, Y., Li, Z., Liu, J., He, Z., Huang, Y., & Li, D. (2014). Word Vector Modeling for Sentiment Analysis of Product Reviews. In Natural Language Processing and Chinese Computing (pp. 168-180). Springer Berlin Heidelberg.