I am trying to implement an optimization procedure in Python using BFGS and L-BFGS in Python, and I am getting surprisingly different results in the two cases. L-BFGS converges to the proper minimum super fast, whereas BFGS converges very slowly, and that too to a nonsensical minimum.
QUESTION: From my readings, it seems to me that BFGS and L-BFGS are basically the algorithm (quasi-Newton methods), except that the latter uses less memory, and hence is faster. Is that true? Otherwise, if they are more different, then how so?
Ultimately, I want to figure out if the difference in performance is due to some differences in the actual algorithms, or due to their implementation in the python SciPy modules.
EDIT: I am adding some data to support my claims of divergent behaviour from the two algorithms.
RUNNING THE L-BFGS-B CODE
* * *
Machine precision = 2.220D-16
N = 147 M = 10
This problem is unconstrained.
At X0 0 variables are exactly at the bounds
At iterate 0 f= 2.56421D+04 |proj g|= 1.19078D+03
At iterate 1 f= 2.12904D+04 |proj g|= 1.04402D+03
At iterate 2 f= 1.49651D+03 |proj g|= 2.13394D+02
At iterate 3 f= 6.08288D+02 |proj g|= 9.85720D+01
At iterate 4 f= 2.91810D+02 |proj g|= 6.23062D+01
...
At iterate 142 f= 3.27609D+00 |proj g|= 8.80170D-04
Time taken for minimisation: 36.3749790192
*** BFGS code ***
At iterate 1, f= 21249.561722
At iterate 2, f= 15710.435098
At iterate 3, f= 15443.836262
At iterate 4, f= 15386.035398
At iterate 5, f= 15311.242917
At iterate 6, f= 15211.986938
At iterate 7, f= 15022.632266
...
At iterate 524, f= 67.898495
...
Warning: Desired error not necessarily achieved due to precision loss.
Iterations: 1239
Time taken: 340.728140116