3
$\begingroup$

Several papers(1 (originator), 2, 3) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand).

I haven't been able to get clear in my head what this means the actual final layer and output/labels of the neural network are.

For (plain) softmax the activation function is the softmax function: $$\mathbf{\hat{y}}=\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}$$

and the loss (error) function is cross entrypy $$C(\mathbf{\hat{y}},\mathbf{y})=\sum_{k=1}^K-\mathbf{y_k}\times \log{\mathbf{\hat{y}_k}}$$ where y is "one-hot" -- all zeros except a 1 for the index matching the class (this lead to effient implementation, if you know the class indexes).

For Hierachical Softmax: What is the form of the label y, the activation function $\sigma(\mathbf{z})$ and the loss (error) function $C(\mathbf{\hat{y}},\mathbf{y})$

I am starting the suspect that the label is a Binary code for the class, eg a Huffman code, and the activation function is simply sigmoid (or tanh) and the loss is just squared error.

Is that all there is too it?

Or is it infact done with a multilayer network, in some way? (Obviously you can't stack softmax layers as inputs to softmax layers).

Implementations

There are quiet a few implementations around, but I find all of them hard to follow.

  • Word2Vec in C, and Gensim in Python.

    • I'm not great at understanding C -- too many clever tricks (like using 1D indexing + offsets on 2D arrays), and the python harks close to the C (it is an enhanced translation).
    • There are two linked articles A, B which go someway towards explaining the C code.

  • A very different python (Cython actually)

  • A even more different Python (Theano) implementation. This one is not truly Hierarchical soft-max as it only has two layers.

Papers

  1. Morin, F., & Bengio, Y. (2005, January). Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics (pp. 246-252).
  2. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119).
  3. Wang, Y., Li, Z., Liu, J., He, Z., Huang, Y., & Li, D. (2014). Word Vector Modeling for Sentiment Analysis of Product Reviews. In Natural Language Processing and Chinese Computing (pp. 168-180). Springer Berlin Heidelberg.
| cite | improve this question | |
$\endgroup$
  • $\begingroup$ I didn't understand. You already answered your question? The activation function is softmax, cost function is cross entropy and labels are one-hot. Labels are not related with Huffman coding. Huffman coding is simply using shorther discripters to more commonly occuring patterns. What are you asking exactly? Could you give a reference to the papers you mentioned about? $\endgroup$ – yasin.yazici Jun 18 '15 at 18:34
  • $\begingroup$ @yasin.yazici: Hierachical Softmax, not ordinary softmax. I will add links to the papers when I get back to the lab. $\endgroup$ – Lyndon White Jun 19 '15 at 0:19
  • $\begingroup$ @Raphael Thanks for the tip, someone (maybe me, when less busy), should make a script to this. $\endgroup$ – Lyndon White Jun 25 '15 at 8:48
  • $\begingroup$ Since there is no single correct format, and no general way to obtain robust URLs, I'm not sure that's a feasible quest. :) $\endgroup$ – Raphael♦ Jun 25 '15 at 10:06
  • $\begingroup$ I think this is a good explanation youtu.be/B95LTf2rVWM $\endgroup$ – Sam H. Oct 10 '18 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.