0
$\begingroup$

Take a Vanilla RNN represented by the function $h_t = f(h_{t-1}, x_t)$, how do you determine $h_0$?

Edit: This answer over on the stats page has helped.

$\endgroup$
  • $\begingroup$ Please write out what is relevant from the link. Links go stale, and not everybody is inclined to click around. Make it easy on the answerer! $\endgroup$ – vonbrand Aug 11 at 21:19
1
$\begingroup$

There are a few different ways. One way is to always give it a zero vector, or some other constant that means "new sequence". Another way is to train the initial state vector just like you train any other weight: initialize it randomly, then apply gradient descent during training. The former is easier and faster, but the latter can sometimes make the RNN more effective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.