I have this dataset, and I am using y = (a * x^n) / (b + x^n) Hill function as the model, where a is the limit of the Hill curve, b is the point at which a/2 is reached (for n = 1) and n is the cooperativity or steepness of the curve.

Currently, I am storing all x,y values and computing the parameters a,b,n using black-box optimization to minimize the least-squares error. (I currently use scipy.optimize.curve_fit, a standard optimizer that can minimize an arbitrary objective function. In my case, the objective function is the total least-squares error.).

When new data points come along, I re-calculate the parameters with the old+new data, i.e. I append the new data to the old tuple of x,y values and get the new parameters.

Is there a way to update the parameters of the model without storing all of the previous old data points, once the initial parameters are obtained from the previous data points? I am looking for some kind of on-line algorithm to tackle the problem, but have no clue how to go about it. I read a blog-post that shows how do this for polynomial regression, but as the power of x which is n is a parameter to the model, I have no idea how to extend the approach to my specific function.

I am looking for a solution like so: I fit the curve to the first 1000 data points and have my parameters. Next, I discard some or all of the old data. Then, when I see the 1001st point I simply update my parameters and plot the curve again and so on for every new data point.

  • $\begingroup$ @YuvalFilmus, does the question in it's current form work for CS Stack Exchange? $\endgroup$
    – neo4k
    Aug 16 '17 at 0:15
  • $\begingroup$ Yes, it's on topic. $\endgroup$ Aug 16 '17 at 6:03
  • $\begingroup$ Check my edit to see if it accurately represents your problem. (And welcome to CS.SE! I think the question is on-topic and suitable here, after the edits to focus on an algorithm rather than an implementation. I hope someone will be able to help you.) $\endgroup$
    – D.W.
    Aug 16 '17 at 6:27
  • $\begingroup$ Must $n$ be an integer ? If so, a dumb engineer's method (parse that either way) is to take the best of $opt(n-1), opt(n), opt(n+1)$ on new + "some" old data. (I believe scipy.optimize.least_squares can be run incrementally, starting from previous $x \ grad \ jac$.) $\endgroup$
    – denis
    Feb 22 '18 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.