What is Temperature in LSTM (and neural networks generally)?

Question

One of the hyperparameters for LSTM networks is temperature. What is it?

Answer 1

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s Magenta implementation of LSTMs, temperature represents how much to divide the logits by before computing the softmax.

When the temperature is 1, we compute the softmax directly on the logits (the unscaled output of earlier layers), and using a temperature of 0.6 the model computes the softmax on $\frac{logits}{0.6}$, resulting in a larger value. Performing softmax on larger values makes the LSTM more confident (less input is needed to activate the output layer) but also more conservative in its samples (it is less likely to sample from unlikely candidates). Using a higher temperature produces a softer probability distribution over the classes, and makes the RNN more “easily excited” by samples, resulting in more diversity and also more mistakes.

Neural networks produce class probabilities with logit vector $\mathbf{z}$ where $\mathbf{z} =$$(z_1,\ldots,z_n)$ by performing the softmax function to produce probability vector $\mathbf{q} = (q_1,\ldots,q_n)$ by comparing $z_i$ with with the other logits.

$q_i = \frac{\exp{(z_i/T)}}{\sum_j\exp{(z_j/T)}}\tag{1}$

where $T$ is the temperature parameter, normally set to 1.

The softmax function normalizes the candidates at each iteration of the network based on their exponential values by ensuring the network outputs are all between zero and one at every timestep.

Temperature therefore increases the sensitivity to low probability candidates. In LSTMs, the candidate, or sample, can be a letter, a word, or musical note, for example:

For high temperatures (${\displaystyle \tau \to \infty }$), all [samples] have nearly the same probability and the lower the temperature, the more expected rewards affect the probability. For a low temperature (${\displaystyle \tau \to 0^{+}}$), the probability of the [sample] with the highest expected reward tends to 1.

- from Wikipedia article on softmax function

Reference

Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015). arXiv