4
$\begingroup$

It seems like an RNN has to have the ht-1 needs to be the same size as the input vector since they're being added to one another, but if you're doing something like modeling to another language or classification, how would you handle this? I'd imagine you'd just have a fully connected layer that would predict the output, but I was wondering if there is a standard convention?

$\endgroup$
2
$\begingroup$

So Basically what happens is the input x is multiplied by a hidden layer U to yield a state vector s, which transforms it to a new dimensionality. Then The recurrent layer W is always a square matrix to maintain the dimensions of s. Lastly, the output matrix V is multiplied to update the matrix to the output dimensionality.

So if you had a vector that was 5 dimensions, and you wanted to multiply it into a state vector of 20 dimensions and an output vector of 2 dimensions, your dimensions would be:

x is 5x1
s is 20x1
y is 2x1

U is 20x5
W is 20x20
V is 2x20

and your formula is:

s = Ux + W*s_(t-1)
y = Vs
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.