# Training a neural net without labels in Reinforcement Learning

I am trying to dig further into machine learning and I am making a program to play a game as a start. I have created a game that is based on the mobile game Flappy Bird and can be generalized to the following:

• Actions $A = \{\text{no action}, \text{tap}\}$
• States $S \subset \mathbb{R}^3$ such that $s \in S$,
• $s_1$ is the horizontal distance to the center of the nearest forward gap
• $s_2$ is the vertical distance to the center of the nearest forward gap
• $s_3$ is the vertical velocity ($\text{unit}/\text{sec}$)

As a starting point, I have implemented a naive Q-Learning algorithm with positive rewards whenever it passes through a gap successfully. Everything went well and the q-table ($f(s): S \to A$) was adjusted accordingly to the rewards.

Now I would like to train a neural net to do the same. One problem that existed with the Q-Learning approach is that the algorithm does not take locality into account. I had to also discretize the feature space into very huge chunks which is not optimal. I am hoping that a neural net can solve these issues.

I am familiar with performing basic classifications with neural nets, and so I am trying to approach this problem similarly. I would just take a state (a vector of three real numbers) as input and an action as output (one neuron per action and take the one with the highest probability). The problem is, I do not have labels for training! The only thing that I get from the game is reward.

Is a neural net a good choice for this type of problem? How should I train or design such network?

What you are looking for is DQN: Deep Q-Network.

You want to have a look at the example code, but the idea is as follows:

• Input: The observation
• Output: Reward - the action can either be in the input or better one reward-prediction per possible action, if the action is discrete.

For final states, you know the true reward. Hence your target is clear (see line 112). But if this is not the case, you need to adjust the target to the observed reward and the discounted expected reward (line 113ff).

Extremely important: The output layers activation function. Don't use softmax, as it is unlikely that your rewards sum up to 1. Don't use ReLU if the reward can be negtive. Simply using a linear output makes most sense in most cases.

The whole point of reinforcement learning is that it gives you a way to train a machine learning algorithm, without having labels for training. It provides a way to obtain labels; then you use some existing ML method to build a policy. You can use any ML method you like, including a neural network -- neural networks are often chosen for this purpose, because they seem to work well in many situations.

So, I suggest that the best answer to your question is going to be: use reinforcement learning with neural networks. See, e.g., https://karpathy.github.io/2016/05/31/rl/ for an overview of how you could do that.

See also Are neural networks a type of reinforcement learning or are they different? for an overview of the relationship between reinforcement learning, supervised learning, and neural networks.

• The links you provided are great resources, thanks! – Derek 朕會功夫 Mar 11 '18 at 3:35