5
$\begingroup$

I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.

Typically, it averages becoming more effective, eventually perfectly solving the environment (500/500 reward for several hundred games in a row) but then starts to regress. Typically somewhere in the range of 100~150/500 total reward. This seems to happen regardless of learning rate, but I haven't tested with very low learning rates (less than 0.01) because of the amount of time it takes to train.

Can anyone tell me why this happens? I can't seem to find any literature on it, but perhaps I just don't know the name.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Please post your working code, or e.g. a github reference. $\endgroup$
    – J_H
    Nov 1, 2017 at 20:50
  • $\begingroup$ @D.BenKnoble Overfitting in reinforcement learning isn't the same as overfitting in supervised learning, and as such not relevant here. $\endgroup$
    – Omegastick
    Dec 27, 2019 at 4:02

2 Answers 2

Reset to default
0
$\begingroup$

There seems to be some kind of overfitting happening.It happens when noise and fluctuation in data is learned as concept by your Neural Network Model.

https://en.wikipedia.org/wiki/Overfitting

Improve this answer
$\endgroup$
0
$\begingroup$

It's been a couple of years since I asked this question, and my knowledge has advanced quite a bit in that time, so I figured I'd answer this as best I can.

This is, as far as I know, still an open research problem. Policy-gradient agents diverging after converging for a while is a known problem, and there are a variety of hypotheses around it. KL-divergence constraining techniques (PPO, TRPO) seem to help stabilize a bit, so it's likely something to do with KL-divergence.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.