Reinforcement learning without self-play

Question

Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?

For example, let's say that I want to build a TRPG AI, which involves natural language and physical interaction between heterogeneous players. I can't do self-play this game so many times. But I happen to have a lot of match log of past human games.

Can I achieve higher score than humans using reinforcement learning using these logs (maybe like Experience Replay)?

Answer 1

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to generate many more training examples on which to train a policy by selecting new action combinations. This may result in unexpected behaviour if you do not have sufficient examples which is quite likely to occur unless you have a very large training set.

Reading up on Actor-Critic reinforcement learning may help you here.

Answer 2

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a fruitful manner.