I'm trying to generate some history-dependent model with machine learning, whose underline physical model has a clear definition of its "internal state variable" (a state derived from historical inputs) and how this variable interacts with the inputs to get the outputs. Mathematically this reads: $y_t=f(x_t,h_t)$, $h_t=g(x_1, ..., x_{t-1})$ .

Intuitively, this can be realized by RNN(or LSTM/GRU). Now I wonder if we can add the supervision on state variables to the loss function to improve prediction. For instance, we extract the hidden states $H$ output from our RNN and pass them through another branch of feed-forward network to predict the internal state variables $h$. The loss function then has two parts, one is for model outputs $\sum (y-\hat{y})^2$, and the other is for model internal states $\sum(h-\hat{h})^2$.

Any insight toward the following questions are welcomed:

  • Will this approach work? (Does it make sense?)
  • Potential issues?

I'll also be really grateful if anybody can indicate closely related works in the answers! Thanks in advance!


There are several examples of performing supervision on hidden states for history dependent models.

Latent variables

Loss functions are commonly applied to internal states such as latent variables in variational autoencoders (VAEs).

Hidden states

Supervision directly on hidden states, as opposed to latent states in RNNs, has been performed for imposing sparsity, as in [1]:

In order to impose sparsity, we introduce an L1-norm loss term over the output gate at layer $l$ as

$$\mathcal{L}^S(o^l(x,\theta)) = \sum_{1 \leq b \leq B} \sum_{1 \leq > t\leq T} | o_t^l(x^{(b)})|.$$

Autoregressive Neural Networks

Additional, auxilliary latent variables have been used to increase the flexibility of variational inference. An example of this is given in [2] where hidden state $\mathbf{h}$ and latent state $\mathbf{z}_{t-1}$ are fed into an autoregressive neural network, with final output $\mathbf{z}_t$.


1 Learning Sparse Hidden States in Long Short-Term Memory, (paper)

2 Improved Variational Inference with Inverse Autoregressive Flow, (arxiv)


Your physical model looks dubious. It is a bit unusual and might not be a perfect match for a RNN/LSTM. I am hoping that your physical model is actually $y_t=f(x_t,h_t)$, $h_t=g(x_{t-1},h_{t-1})$. If so, then this is a good fit for the kind of supervision on internal states that you plan to do -- and the answer to your question is yes, you can do this, and it is a sensible thing to do.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.