# Why does the effectiveness of my reinforcement based neural network recede after a while?

I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.

Typically, it averages becoming more effective, eventually perfectly solving the environment (500/500 reward for several hundred games in a row) but then starts to regress. Typically somewhere in the range of 100~150/500 total reward. This seems to happen regardless of learning rate, but I haven't tested with very low learning rates (less than 0.01) because of the amount of time it takes to train.

Can anyone tell me why this happens? I can't seem to find any literature on it, but perhaps I just don't know the name.

• Please post your working code, or e.g. a github reference. – J_H Nov 1 '17 at 20:50
• @D.BenKnoble Overfitting in reinforcement learning isn't the same as overfitting in supervised learning, and as such not relevant here. – Omegastick Dec 27 '19 at 4:02

There seems to be some kind of overfitting happening.It happens when noise and fluctuation in data is learned as concept by your Neural Network Model.

https://en.wikipedia.org/wiki/Overfitting

It's been a couple of years since I asked this question, and my knowledge has advanced quite a bit in that time, so I figured I'd answer this as best I can.

This is, as far as I know, still an open research problem. Policy-gradient agents diverging after converging for a while is a known problem, and there are a variety of hypotheses around it. KL-divergence constraining techniques (PPO, TRPO) seem to help stabilize a bit, so it's likely something to do with KL-divergence.