Neural network latent (or pre-activation) representations are the weighted sums of inputs to neurons in hidden layers before applying an activation function. The vector of representations of neurons in a layer $l$ can be represented as $\mathbf{z_l}$ where $l$ can be, for example, a fully-connected or convolutional layer. At a given time step $n$, the layer representations $\mathbf{z_{l,n}}$ are often fitting a Gaussian-like bell distribution, for example:

Preactivation Gaussian distribution (image source)

This is, I presume, especially likely if the outputs of layer $l-1$ are batch normalized.

It would be interesting to see if moments of the distribution of $\mathbf{z_{l,n}}$ (mean, variance, etc.) can be useful signals for training dynamics.

My question is about finding an explanation or mention of the distribution of representations in the literature. Is there any reference in the literature supporting the observation that the representations of layers during training are normally distributed, or explaining what non-normally distributed layer representations indicates in terms of training dynamics?

  • $\begingroup$ I'm confused about what distinction you are drawing. You say that pre-activation representations often fit a Gaussian, and then you ask if pre-activations are distributed in a particular way. What's the difference? Perhaps you can be more precise about what you mean by things like "before batch distribution" and "within a layer" and "pre-activation". Can you write out a mathematical representation of a layer and indicate specifically which value you are interested in? $\endgroup$
    – D.W.
    Nov 27 '19 at 21:46
  • $\begingroup$ @D.W. Thanks for your comment. I corrected "batch distribution" to "batch normalization" and wrote out a mathematical representation. $\endgroup$ Nov 27 '19 at 23:00
  • $\begingroup$ I'm confused what your question is, then. I'm not sure what is meant by a "signal for training dynamics". I do not expect batch normalization to affect the distribution; if it has a Gaussian distribution before batch normalization, it will afterwards as well. $\endgroup$
    – D.W.
    Nov 28 '19 at 3:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.