So we are calculating the loss

$$J(\theta) = -\frac{1}{T}\sum_{t=1}^T\sum_{-m \leq j \leq m} \log P(w_{t+j}|w_t;\theta)$$

and to do this we need to calculate

$$P(o|c) = \frac{\exp(u_o^Tv_c)}{\sum \exp(u_w^Tv_c)},$$

which is computationally inefficient. To solve this we could use the hierarchical softmax and construct a tree based on word frequency. However, I am having trouble on how we could get the probability based on the word frequency. And what exactly is the backprop step if using hierarchical softmax?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.