I was looking at a tutorial for recurrent neural networks in Python, and I have a question in regards to multiplying matrices of different sizes. Specifically, why does S[t] have 100 elements in it?

s[t] = np.tanh(self.U[:,x[t]] + self.W.dot(s[t-1]))

Earlier in the tutorial, the author lists the dimensions for each variable:

\begin{aligned} x_t & \in \mathbb{R}^{8000} \\ o_t & \in \mathbb{R}^{8000} \\ s_t & \in \mathbb{R}^{100} \\ U & \in \mathbb{R}^{100 \times 8000} \\ V & \in \mathbb{R}^{8000 \times 100} \\ W & \in \mathbb{R}^{100 \times 100} \\ \end{aligned}

From how I understand the above line of code, it multiplies U by x[t] and adds it to the product of W and s[t-1], then computes tanh for each element.

Sources like this say that you cannot add matrices of different dimensions, however that seems to be what is happening here (because multiplication is just repeated addition). In fact, it seems that U is 2D and x[t] is 1D. How are these added? Also, how is the sum of the two products then 100 elements?


Matrix-by-matrix multiplication is very different from scalar-by-scalar. It has no connection to repeated addition, and in fact isn't even commutative: it's entirely possible that $AB \neq BA$. It's only called multiplication because it has some similar properties to repeated addition of scalars.

Matrix multiplication requires that the inner dimensions match. Nothing more, nothing less. You can multiply an (a×b) matrix by a (b×c) matrix, for instance, and the result is an (a×c) matrix. For the full details, Wikipedia has a good summary.

In addition, that code isn't adding or multiplying $U$ and $x[t]$. Rather, it's removing one dimension from $U$, taking only the columns (or rows depending on your definitions) which correspond to 1s in $x[t]$.

  • $\begingroup$ OK, that makes sense. Is there any way to achieve similar results on a per-element basis? Meaning avoiding using matrix multiplication. P.S. Because x[t] is a one-hot vector, it is equivalent to multiplying U and x[t]. $\endgroup$
    – APCoding
    Jun 10 '18 at 21:41
  • $\begingroup$ @APCoding Selecting only a single column/row from a matrix is, it turns out, very different from multiplying it! Multiplying by a one-hot vector zeroes out some columns/rows but doesn't remove them. So the resulting dimension is different. $\endgroup$
    – Draconis
    Jun 10 '18 at 21:59
  • $\begingroup$ @APCoding All modern machine learning algorithms are based on matrices and vectors in some way, especially neural networks. Keeping track of every weight and bias individually is a Very Bad Idea given the scale of even tiny toy networks. If you want to work with neural networks, you should read up a bit on vectors and matrices and how they interact. $\endgroup$
    – Draconis
    Jun 10 '18 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.