How much memory do we need to train a multilayer perceptron?

I've started to figure this out myself, but I'm stuck.

I have one-layer MLP. Each training example is a vector of 100 real numbers in float32. There are 1000 training examples. The hidden layer is 50 units. The output layer is 10 units.

Karpathy's excellent blog gives me a starting point.

Weights and biases from input to hidden: (100 x 50) + 50 = 5050

Weights and biases from hidden to output: (50 x 10) + 10 = 510

Total = 5560 parameters

To store the parameters, I'll need (5560x4)/(1024^2) = 0.019 MB

I have minibatches of size 100, so I need to read 100 examples at a time into memory:

(100 x 100)x4/(1024^2) = 0.038 MB

I think I'll also need to store:

  • the gradients of the activations
  • the gradients with respect to the weights (?)
  • the step cache (since I'm using momentum)

I'm using rmsprop as the optimizer.

Do I need to store the gradients of the activations AND the gradients of the weights? How much memory do these require? How about the step cache? Am I missing anything?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.