How much memory do we need to train a multilayer perceptron?
I've started to figure this out myself, but I'm stuck.
I have one-layer MLP. Each training example is a vector of 100 real numbers in float32. There are 1000 training examples. The hidden layer is 50 units. The output layer is 10 units.
Karpathy's excellent blog gives me a starting point.
Weights and biases from input to hidden: (100 x 50) + 50 = 5050
Weights and biases from hidden to output: (50 x 10) + 10 = 510
Total = 5560 parameters
To store the parameters, I'll need (5560x4)/(1024^2) = 0.019 MB
I have minibatches of size 100, so I need to read 100 examples at a time into memory:
(100 x 100)x4/(1024^2) = 0.038 MB
I think I'll also need to store:
- the gradients of the activations
- the gradients with respect to the weights (?)
- the step cache (since I'm using momentum)
I'm using rmsprop as the optimizer.
Do I need to store the gradients of the activations AND the gradients of the weights? How much memory do these require? How about the step cache? Am I missing anything?