My objective function, f, is complicated and embodies several disparate constraints I want my simulation to optimize simultaneously. So I can't really even assume it's continuous; it is probably close, in that small changes in the parameter space won't produce big shifts in f, but the gradient, etc. will be undefined all over the place. Also I'm interested in solutions that won't require me to derive the gradient, hessian, etc. every time I make a change to the model.
In my corner, I've got a Tesla K20X. I know that any reasonable N-dimensional grid around the region of likely minima will contain numerous cells that will be nowhere close to optimal. That is, the error of most combinations of the N free parameters will be essentially infinite.
So far I've just been using a standard unconstrained search function (Matlab's fminsearch) on the CPU from numerous different starting points. This takes too long, and I'm worried about local minima (which will always be a concern, but perhaps less so with a better approach). Aside from just implementing that on the GPU (each thread doing its own fminsearch, which seems wasteful), I've thought of a kind of divide-and-conquer approach--evaluate the model at a relatively coarse N-dimensional grid of points in parameter space, throw out those cells with the most error, refine the grid, re-evaluate keeping the total number of evaluations about the same, and keep iterating until the error falls below threshold/won't shrink any more. What else is there? My searching has mainly come up with algorithms that expect you to do some math on the equations, but for the models I eventually plan to run (input-driven dynamical systems with dozens of state variables) this looks to be infeasible.