I am training a Variational Autoencoder (type of convolutional neural network), and have been plotting cost over time. The result is a noisy curve, shown here:Cost over Time

I would like to write a function that gives the probability that the cost function has bottomed out, i.e. that its slope going into the future is zero. Due to the noise, I can't just take the derivative of the cost function, as it oscillates stochastically. I could take the derivative of the moving average of the cost function, but it would be unclear what window size to use, and regardless a single window size would be useful for only part of the line (the noise tends to increase over time).

Is there some mathematical function/theorem I can use to take the local noise of the curve into account, and output a range/distribution of probable slopes? I would imagine the output being something like a histogram with the most likely slope at the middle and increasingly less-likely slopes off on the sides.

  • $\begingroup$ Probably I do not understand your question (or misread what is the objective), but why do you want to use noise and output potential slopes? If you do lowpass filter and extrapolate the result, you will get as accurate output as fitting curve was able to (being the proper model). The marching average may also work, but here the longer window gives smoother result. $\endgroup$ – Evil Jun 4 '17 at 19:58
  • $\begingroup$ Also, are you sure that the slope being closer to 0 is enough information? Looking at your graph above, it appears to level out to nearly 0 slope, then the slope becomes more negative again. $\endgroup$ – user1118321 Jun 5 '17 at 3:31
  • $\begingroup$ Unfortunately, the slope being close to zero isn't REALLY enough information to be sure - but it's an OK indicator in any event. I wouldn't be making any final decisions on whether or not to terminate training with the output of the above desired function, but rather, using the output as part of a larger deciding process. $\endgroup$ – Matt Billman Jun 6 '17 at 11:54

A moving window (with some reasonable window size) is probably hard to beat. I don't know of any parameter-less solution.

Another possible approach could be to estimate the slope of the last 10% of the data, and compare this to some threshold. (So, the 1000th data point uses the average of points 900-1000. The 2000th data point uses the average of points 1800-2000. And so on.) This way, if it does flatten out, then if you continue for another 10% longer, you'll reach a point where the last 10% is all flat and the procedure will likely terminate. This may do 10% more iterations than "necessary", but that might be acceptable given that in return you get a procedure that doesn't require a fixed window size.

However, choosing a threshold might be a bit tricky. I'm not sure how to do that in a reasonable way.

Given a bunch of data points, you can compute a least-squares fit for the regression line through those points, and find an estimate of the slope of that line. You can also compute a confidence interval for that slope. This will give you a range of plausible slopes.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.