I stumbled upon a statement like the following, in the context of prediction (classification) with a neural network: "If we predict the class of a given data point $x$ with random weights, the output will be random and thus the classification will be wrong in most cases." This is true, no doubt about that. But what would be the distribution of the output in the following example: input dimension=1, 0 hidden layers, weight $w$ distributed $N(0, 1)$ (standard normal distribution), activation function=sigmoid$=\sigma(x)=\frac{1}{1+exp(-x)}$, and input=1? We would have to compute the distribution of $\sigma(w)$. The expected value is, which seems reasonable, $\frac 12$, because $\int_\mathbb{R} \frac{exp(-\frac 12 x^2)}{(1+exp(-x))\sqrt{2\pi}}dx=\frac 12$, according to Wolfram Alpha. The variance is not as easy anymore, with an approximate 0.043379. Does anyone of you know how to compute the complete distribution?



  • $\begingroup$ This question is probably on-topic here, but may be more suitable for Cross Validated. Consider requesting migration if your question doesn't get answered in a couple of days. $\endgroup$ – Discrete lizard Jan 22 '18 at 14:08
  • $\begingroup$ It´s probably just concatenating with the inverse of the sigmoid, which is luckily monotonically increasing: $P(\sigma(w)<=z)=P(w<=\sigma^{-1}(z))$. $\endgroup$ – Leon Jan 22 '18 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.