Machine learning and test split for time series data

Question

I have used different machine learning algorithms to predict solar panels' power output. There are ten independent features for weather data. In all models, I set time as an index and have used the following rule for splitting the data :

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.18, random_state=42)

With the mentioned splitting rule, all models perform well on the testing set, but I get the worst accuracy when not using a random rule for the testing set.

Does it make sense to use a random rule for splitting the dataset into training and testing sets? Or do I have to use only the latest data (20%) for the test split?

Answer 1

In most contexts, random shuffling is absolutely required. If you do not randomly split the dataset, you might get misleading results. For instance, imagine that we have a dataset of patients, and we train a classifier on the first 60% and test it on the last 40% of patients. Well, if the patients happen to be ordered in the dataset in a particular order -- say, youngest to oldest -- then you might end up with a classifier that works well on young folks and poorly on old folks, and thus that achieves very poor accuracy on the test set (which is all old folks) even though it is better than that in practice.

In your situation, the obvious approach is to randomly shuffle. This assumes that there is no change over time in the underlying data distribution, and is appropriate if so. If that assumption is not correct (e.g., partway through the dataset you replaced the solar panels with new ones), then you'll need to do something more complex.