Neural network latent (or pre-activation) representations are the weighted sums of inputs to neurons in hidden layers before applying an activation function. The vector of representations of neurons in a layer $l$ can be represented as $\mathbf{z_l}$ where $l$ can be, for example, a fully-connected or convolutional layer. At a given time step $n$, the layer representations $\mathbf{z_{l,n}}$ are often fitting a Gaussian-like bell distribution, for example:
This is, I presume, especially likely if the outputs of layer $l-1$ are batch normalized.
It would be interesting to see if moments of the distribution of $\mathbf{z_{l,n}}$ (mean, variance, etc.) can be useful signals for training dynamics.
My question is about finding an explanation or mention of the distribution of representations in the literature. Is there any reference in the literature supporting the observation that the representations of layers during training are normally distributed, or explaining what non-normally distributed layer representations indicates in terms of training dynamics?
