1
$\begingroup$

I have used different machine learning algorithms to predict solar panels' power output. There are ten independent features for weather data. In all models, I set time as an index and have used the following rule for splitting the data :

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.18, random_state=42)

With the mentioned splitting rule, all models perform well on the testing set, but I get the worst accuracy when not using a random rule for the testing set.

Does it make sense to use a random rule for splitting the dataset into training and testing sets? Or do I have to use only the latest data (20%) for the test split?

$\endgroup$
1
  • 1
    $\begingroup$ Random shuffling the dataset is advisable instead of simply taking the latest test data (or any particular segment of data every time) since the input data might not be given to you uniformly distributed form. $\endgroup$ Jan 27, 2022 at 20:38

1 Answer 1

2
$\begingroup$

In most contexts, random shuffling is absolutely required. If you do not randomly split the dataset, you might get misleading results. For instance, imagine that we have a dataset of patients, and we train a classifier on the first 60% and test it on the last 40% of patients. Well, if the patients happen to be ordered in the dataset in a particular order -- say, youngest to oldest -- then you might end up with a classifier that works well on young folks and poorly on old folks, and thus that achieves very poor accuracy on the test set (which is all old folks) even though it is better than that in practice.

In your situation, the obvious approach is to randomly shuffle. This assumes that there is no change over time in the underlying data distribution, and is appropriate if so. If that assumption is not correct (e.g., partway through the dataset you replaced the solar panels with new ones), then you'll need to do something more complex.

$\endgroup$
2
  • $\begingroup$ Thanks for your comment. Do you think we can randomly choose 20% of the dataset for the testing set and then for cross-validation use a KFold technique with shuffling the data to check whether the results are acceptable or not? $\endgroup$ Jan 28, 2022 at 9:21
  • 1
    $\begingroup$ @graphicart86, yes $\endgroup$
    – D.W.
    Jan 28, 2022 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.