Method for combining derivative free optimization results of different data inputs

I am working on an algorithm that has multiple fixed parameters. The algorithm analyzes time series data and spits out a number. The fixed parameters need to be such that this number is as small as possible.

What I found, is that when optimizing the parameters for a specific time period, these parameters don't necessarily work well when used on another time period.

The way I see it, is that there are two possible solutions to this problem:

1. use a longer time period when optimizing the parameters
2. find a method of combining the optimal parameters for different time periods, such that these "averaged" parameters work well on all time periods

Option 1. would be incredibly expensive in terms of computational time. And although it makes intuitive sense that this should fix the problem, I am not sure that this would indeed be the case.

Option 2. reminds me of training neural networks, where one would feed in a large number of "data points" and somehow take a (weighted) average of the results to find a set of parameters that work well for all data points. Unfortunately, I know very little to nothing about the algorithms used for this kind of optimization/learning.

Any help or suggestions are greatly appreciated. Please let me know if there is anything you'd like me to expand upon.

Thanks!

• my loss function is not an actual mathematical function, so I don't know exactly what it looks like, this also means that I can't compute the gradient.
• The optimization method I currently use is differential evolution using the best1bin strategy.
• The reason the predictions fail is because of overfitting.
• There are ~10 parameters
• One iteration takes about 5-8 seconds
• This question might be more appropriate for Data Science. If you ask it over there, please remove it here, to prevent duplicates. Jun 3, 2020 at 9:43
• This might be hard to answer without knowing what your loss function looks like, how the predictions depend on the parameters, whether you can compute a gradient, what optimization method you're currently using, and why the predictions fail on future times. I suggest reading about SGD, overfitting, and regularization.
– D.W.
Jun 4, 2020 at 2:25
• @D.W. To answer your questions: my loss function is not an actual mathematical function, so I don't know exactly what it looks like, this also means that I can't compute the gradient. The optimization method I currently use is differential evolution using the best1bin strategy. The reason the predictions fail is because of overfitting. Thank you for your suggestions, I'll read into them! Jun 4, 2020 at 9:04
• Thanks. Can you edit your question to incorporate that information into the question, so it reads well for someone who encounters it for the first time and so that people don't have to read the comments to understand what you are asking?
– D.W.
Jun 4, 2020 at 17:54
• How many parameters are there? There are a number of gradient-free optimization methods.
– D.W.
Jun 4, 2020 at 17:56

You say you are overfitting. The usual solution is to use more training data and/or regularization.

The easiest solution is probably going to be to train on all data, not just a limited time window. Perhaps you can find a better mathematical optimization algorithm. You don't give any details on the optimization algorithm you are using or how you compute the loss function; perhaps using a stochastic version (e.g., analogous to SGD instead of GD) would provide better efficiency. You could also consider Bayesian optimization, which constructs a differentiable model to estimate/predict the output of your loss function, and then uses SGD on it. Another option is to use gradient-based methods and estimate the gradient using finite differences; if you have 10 parameters, you can estimate the gradient by evaluating the function 11 times on 11 different inputs. Then you could combine that with gradient descent.

Or perhaps you can use regularization; for instance, if you have a prior on parameters, you might be able to choose a suitable regularization method. I suspect it's going to be hard to say without getting into the specifics.

• Yes it seems like using more data here would indeed be the way to go. Thank you for your suggestion of using Bayesian optimization. It sounds very interesting, I will definitely look into it Jun 5, 2020 at 10:07