1
$\begingroup$

Most of the discussion about RNN and LSTM alludes to the varying ability of different RNNs to capture "long term dependency". However, most demonstrations use generated text to show the absence of long term dependency for vanilla RNN.

Is there any way to explicitly measure the time dependency of a given trained RNN, much like ACF and PACF of a given ARMA time series?

I am currently trying to look at the (Frobenius norm of) gradients of memories $s_k$ against input $x_l$, where $l\le k$, summed over training examples $\{x^i\}_{i=1}^N$ - $$\text{Dep}(k,l):=\sum_{i=1}^N \big\|\frac{\partial s_k}{\partial x_l}(x^i)\big\|_F$$ I would like to know if there are more refined or widely-used alternatives to this prototype.

I am working with time series so I treat the inputs $\{x_t\}$ as realization of a random process $\{X_t\}$, thus by "current" I mean $x_i,s_i$ for some fixed $i$, "the past" I mean $\{x_j\}_{j=1}^{i-1},\{s_j\}_{j=1}^{i-1}$ and "time" I mean the index $t$.

I guess that the "long-term dependency" in literature refers to the sensitivity of the current memory $s_k$ w.r.t. past inputs $\{x_j\}_{j=1}^{k-1}$, hence the prototype I formulated.

$\endgroup$
0
0
$\begingroup$

I am not aware of any standard or widely-used metric for this. I think what metric would be appropriate would depend on what you want to use it for.

The issues with RNN is "forgetting". If you feed a long sequence of inputs $x=(x_1,\dots,x_n)$ into a RNN, where $n$ is too large, the problem is that often the final decision is determined by the last few values ($\ldots,x_{n-1},x_n$) and the earliest values ($x_1,x_2,\ldots$) have been "forgotten" and do not affect the final decision. This is undesirable in many settings.

Your metric would be one reasonable way to get a feeling for this. Another reasonable way might be to to feed in an input $x=(x_1,x_2,\dots,x_n)$, then change just $x_1$ to get a new input $x'=(x'_1,x_2,\dots,x_n)$, feed in $x'$, and compare the outputs of the RNN on $x$ vs $x'$; and repeat for many training samples or test samples $x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.