Why does a RNN network output different based on the training sequence

Question

I've set up an RNN LSTM network in Java using DL4J as the library.

I currently have 500 examples of positive text, and 500 examples of negative text.

When I fitness the training data by first training all the negatives and then all the positives, my predictions only favor high positive responses even for things that would be considered negative in my training examples.

And if I reverse it and train positive first and then train negative last everything is favored as a negative in high 80 - 90%.

However, when I train in an oscillating pattern such as: Positive, Negative, Positive, Negative and so on, the predictions become accurate again and similar examples of negative text are picked up relatively accurately and vice versa for positive text.

I've only just started studying machine learning in general but I can't quite find any resources explaining why the sequence of training impacted my model so hugely. Is this normally how an LSTM network should behave or is oscillating the training data the correct approach?

Would the classification of a messages negative intent be something an LSTM RNN network should be used for or should I consider another network?

Answer 1

RNNs are pattern recognition tools. It is't entirely clear to me what it is exactly that you are trying to do, but if you simply intend for it to classify positive and negative messages a regular Neural Network might be better suited.

What your RNN does (if implemented correctly) is learn classifications in the context of the sequence. i. e. It is learning: what are the odds of a given input X being positive/negative knowing the preceding N outcomes.

This should already be giving you some insight on why different orderings of the training data skew the results. If your training set is all positives followed by all negatives then the vast majority of your training set will be teaching the network that a sequence of positives almost certainly indicate a positive message and vice versa.

Building a good dataset is by far the most important part of applying machine learning and I suggest you look for some high level explanations of the different types of ML and some general information on bias in data.

Answer 2

Don't train by first training all the negatives or first training all the positives. Instead, randomize the order in which examples are used during training. Moreover, you should use a different random order in each epoch.

When you use a non-random order, catastrophic forgetting can cause performance to suffer, as you have observed.

This is not specific to RNNs; the same observations apply to neural networks for classification as well.