In RL, in a game situation, usually the agent is trained by playing against itself.

When we should not depend on this self-training, and switch to train the agent with a real or different player agent other than the agent it self?

I mean is there any characteristics of the environment that would make self-play worse choice than playing against an existing player?


Ideally when training an agent you would reach an equilibrium between both player strategies strategies (hopefully a Nash equilibrium). When using only self-play this equilibrium might be different to the one you would reach when competing against a specific player, in particular, taking into consideration that the other player might not be very efficient, may have a different reward structure, etc.

Now, answering your specific question about the environment.. I would say that the critical aspect is the game being played. You should check if you are dealing with a zero-sum game, cooperative vs non-cooperative, etc.

  • $\begingroup$ Thanks for the answer. Can you elaborate on the last paragraph? You say to check whether it's zero-sum or not, cooperative or not, etc. Then what should we do with that information, once we have it? How do the answers to those questions affect whether self-play will be more or less effective? Thank you! $\endgroup$ – D.W. Nov 11 '16 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.