2
$\begingroup$

Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?

For example, let's say that I want to build a TRPG AI, which involves natural language and physical interaction between heterogeneous players. I can't do self-play this game so many times. But I happen to have a lot of match log of past human games.

Can I achieve higher score than humans using reinforcement learning using these logs (maybe like Experience Replay)?

$\endgroup$
  • $\begingroup$ You should be able to train using log data. Just note that it takes a bit of training to get somewhere, so figuring out how to make the most out of your data is key if you want to avoid learning via self play. $\endgroup$ – spektr Nov 1 '16 at 0:49
  • $\begingroup$ Thanks choward. "figuring out how to make the most out of your data" <- Do you mean imporving reward function? $\endgroup$ – meirin Nov 2 '16 at 3:11
3
$\begingroup$

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to generate many more training examples on which to train a policy by selecting new action combinations. This may result in unexpected behaviour if you do not have sufficient examples which is quite likely to occur unless you have a very large training set.

Reading up on Actor-Critic reinforcement learning may help you here.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a fruitful manner.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.