Can deep learning be applied to nonlinear parameter estimation problems?

Question

Suppose $y=f(x,p)$ while $x$ and $y$ are variables and $p$ is the parameter (all in vector form), and $f$ is an explicit nonlinear function. We have:

$$y_1^{(1)} = f(x_1^{(1)},p_1), y_1^{(2)} = f(x_1^{(2)},p_1), ..., y_1^{(n_1)} = f(x_1^{(n_1)},p_1)$$

$$y_2^{(1)} = f(x_2^{(1)},p_2), y_2^{(2)} = f(x_2^{(2)},p_2), ..., y_1^{(n_2)} = f(x_1^{(n_2)},p_2)$$

$$...$$

$$y_i^{(1)} = f(x_i^{(1)},p_i), y_i^{(2)} = f(x_i^{(2)},p_i), ..., y_1^{(n_i)} = f(x_1^{(n_i)},p_i)$$

$$...$$

If given a set of $(x_i^{(1)},y_i^{(1)}), (x_i^{(2)},y_i^{(2)}), ...$, we can estimate the value of $p_i$ by means of optimization such as gradient descent or any other algorithms.

My question is : Can the parameter estimation problem described above, which is conventionally solved by optimization methods, be solved by deep learning?

Since $f$ is an explicit math formula, a very large training set can be easily generated by sampling $x$ and $p$ from some sort of prior distribution.

The difficulties I met during searching the answer on google:

1. Since DL itself is an optimization problem (or parameter estimation problem), what I found is mainly basic tutorials of DL rather than something directly relevant.
2. A similar question on Quora: How can I apply machine learning to parameter estimation problem in control theory? Is there any example study you can suggest?. The only answer to this question is very brief and only provides a 132-page thesis. I scanned the thesis and found that it seemed to solve a rather complicated image processing problem by reinforcement learning. I'm not sure if it is relevant.
3. A similar question on StackOverflow: Can I do nonlinear regression with deep learning ? How?. But the author tried to use deep learning to find the relationship between $x$ and $y$, which is different from my question: the function $f$ is already known, I just want the $p$.

The intuition of solving parameter estimation by deep learning instead of direct optimization could be:

1. Optimization methods such as gradient descent is prone to find a local optima. Maybe DL will have some magic.
2. In practice, the parameter estimation problem may need to be solved many times for different set of $(x,y)$, which requires speed. So conventional optimization methods may be too slow for this. In a real case that prompts me to ask this question, each parameter estimation takes 0.6s by the lsqnonlin function of MATLAB. In contrast, although training a neuron network is time-consuming, getting the output from the input only takes several milliseconds for a pre-trained network.
3. The formula $y=f(x,p)$ may be used in practice for many years and proves to be very effective. The parameter $p$ also has clear physical meanings and can indicate the actual conditions of the system/machine/experiment. Both $x$ and $y$ can be easily obtained. So I want to reserve the formula rather than use a blackbox-like network to predict $y$ from $x$ as most DLs do.

The difficulties I met when solving the problem myself:

1. Suppose the size of the $(x_i,y_i)$ set is $n_i$: $(x_i^{(1)},y_i^{(1)}), (x_i^{(2)},y_i^{(2)}), ..., (x_i^{(n_i)},y_i^{(n_i)})$. The problem is: $n_i$ is not fixed. If $n_i=5$, we may estimate $p_i$ with a great error. And if $n_i=1e6$, we may get a very precise estimation of $p_i$. But in both case we can get an estimation of $p_i$ at last. However, multi-layer perceptron (MLP) or convolution neuron network (CNN) both accept fixed number of inputs. If you do image processing, you even need to resize every image to the same size first. It seems that MLP and CNN don't suit for this question.
2. The $(x,y)$ set has no order as time-series. The 1 and 2 in $(x_i^{(1)},y_i^{(1)})$ and $(x_i^{(2)},y_i^{(2}))$ are just indices, which don't imply that $(x_i^{(1)},y_i^{(1)})$ is necessarily before $(x_i^{(2)},y_i^{(2}))$. So recurrent neuron network (RNN) doesn't seem to be a natural option.

Answer 1

No. Deep learning is not useful or relevant here. Deep learning is for classification. You don't have a classification task. Rather, you have an optimization task: find the parameter value $p$ that causes the best match between the given $x$-values and $y$-values.

Deep learning isn't magic sauce that you can apply to anything you might ever think of and expect it to solve your problem. Your question is an instance of the XY problem. There may be suitable methods to speed up optimization, but you'll probably need to ask a new question and tell us something about the structure/properties of $f$ and the typical range of values for $n_i$.

Answer 2

I am interested in this question as well. And I did some research. It turns out that Tensorflow is able to do the single function optimization using SGD, Adam, etc.

Here is a very detailed tutorial.

http://patwie.com/2017/01/04/optimization_with_tf.html

Since the Nonlinear Regression is kind of minimization problem such as minimizing the sum of squared residual, I guess we are able to feed a single objective function of unknown parameters and all the data into Tensorflow to solve the problem.

I have not tried it yet. I will come back with new results.

However, it will not be deep learning or Neural Network anymore.

It just uses the Tensorflow's power to facilitate the optimization.

update:

I did run a model with nonlinear objective function and it works. And then I am thinking that it is a Neural Network with a nonlinear hidden layer.

1st input layer/data/x -> 2nd hidden layer/nonlinear f(x, theta) -> 3rd output layer yhat

the model is

$$\hat y = f(x, \theta)$$

not a linear model

$$a = wx + b$$

The objective function is the sum of squared error: $\sum[(y- \hat y)^2]$.

The Tensorflow use the forward and backward propagation to solve the problem.

We estimate the $\theta$s instead of $w \quad \text{and} \quad b$.

I will MXnet and other packages to see if the idea works as well.