max, min gradient and other terms in Neural Network

Question

This link contains a demo that trains a Convolutional Neural Network on the MNIST digits dataset in browser.

I am not getting below terms- 1. max, min gradient in each layer. 2.max, min activation in layers

3.Forward time per example

4.Backprop time per example

5. Classification loss

6. L2 (a pooling mechanism??! but what this means?) Weight decay loss

   What are the meanings of above terms according to given context (webpage demo)?

Answer 1

I suggest you to read an introduction to neural networks. I like neuralnetworksanddeeplearning.com, but I can also recommend chapter 4.3 of my bachelors thesis.

1. max, min gradient in each layer

The gradient is like the derivative for high-dimensional functions. Imagine it as an arrow on a landscape. The arrow points into the direction of the steepest ascent, we want to go into the other direction (where the error becomes smallest).

2.max, min activation in layers

The activation is $\sum w_i x_i$ (details and my BS thesis). Hence it is what each neuron returns, before the activation function is applied. Now you have many neurons per layer and hence a set of activations $A$. Hence you can take the min / max of it. I don't know why you would take the min, but the maximum might give you an indicator which neurons are "responsible" for which feature.

For CNNs, a complete feature map is usually considered to be one neuron (see my MS thesis for some more details). In your case, it simply seems to be the maximum feature returned (hence the brightest pixel; see input layer)

3.Forward time per example

Measure how long it takes to process the input of a neural network. (There is also the backward pass for the gradient)