# How do I stop “cheating” in reinforcement learning (MLP+Evo. Algorithm)?

I have a two hidden-layer MLP. I am trying to teach it classification of the sine function. For instance, if there is an [x,y] point above the sine function, the ANN should classify that point as a 1. Otherwise, it should be classified as a 0.

It should end up like this. Points above y=sin(x) should be blue. Points below should be red. The network is just outputting 1s or 0s, so we are plotting the point along with a binary value (0=red,1=blue).

Graphic done with Matplotlib.

I feedforwarded my ANN thousands of random points, calculated the error between desired and output, and used that as fitness for an evolutionary algorithm that would evolve the network weights. The lower the error, the more likely the weights would be bred and preserved for the next round.

Unfortunately, the algorithm found a way to "break" the network. It was able to minimize the error while outputting the wrong values. For instance, one network and weight iteration I evolved outputted 0 for all inputs I would give it, even though I got the error down to 0 when I evolved the weights. It's a beauty. The plot is below.

My Python code is pretty big, so I will spare how I wrote the MLP and show you just a few snippets of relevant code.

Here, we have lists of random points and are feeding them through the MLP. If it is trained properly, the error will be close to 0 because the MLP correctly classifies each point.

    i=0
error = 0
while i < 100:
output1 = feedForward(neuralNet, [xValsAbove[i], yValsAbove[i]]) # feedforward a point that is above the sine function.
output2 = feedForward(neuralNet, [xValsBelow[i], yValsBelow[i]]) # feedforward a point that is below the sine function.

if output1 < 0.99:
error += 1 - output1
if output2 > 0.01:
error += output2
i+=1
return error


So, let's start evolving. Generate a bunch of random weights. Feed them a few thousand random points each in a range of -50 to 50 on both axes. Select the ones with the lowest classification error. Continue. After 50 generations, my error is down to 2 or 3. Looks good. Let's feed it a bunch of test points and see how it did.

???

Can anyone help me here?

• I'm not sure what you're looking for. If it's help debugging your code, that's off-topic, here (and probably impossible without access to all the code). Are you looking for help with your general approach? Something else? (It's possible that I don't know what you're asking because I know nothing about machine learning.) – David Richerby May 9 '16 at 6:07
• "outputted -1 for all" ​ error += output2 ​ ​ ​ ​ – user12859 May 9 '16 at 12:14
• @Ricky Demer Sorry, that -1 should actually be a 0. – Strava Ostetnis May 9 '16 at 14:54
• @ David Richerby No, I don't need help with the code. I am looking for a general solution to the problem and am thinking that some MLP programmers here might have experienced the same or similar. – Strava Ostetnis May 9 '16 at 14:56
• The "general solution to the problem" is "debug your code". ​ ​ – user12859 May 9 '16 at 20:24

## 1 Answer

You are minimizing the wrong loss function. In other words, you have picked a poor objective function.

Look up the standard objective functions that are used for training neural networks for boolean classification tasks: e.g., mean-squared error, or cross-entropy loss. This is covered in standard tutorials. They don't use the function you show. The function you picked looks... odd, so I suspect it is the source of your porblem.

• I found out what was wrong. There is nothing wrong with the error function, but somehow the tuples that store my weights and their respective errors are having their values changed. I have no idea how this is possible with a tuple. – Strava Ostetnis May 16 '16 at 3:36