# Can deep learning be applied to nonlinear parameter estimation problems?

Suppose $y=f(x,p)$ while $x$ and $y$ are variables and $p$ is the parameter (all in vector form), and $f$ is an explicit nonlinear function. We have:

$$y_1^{(1)} = f(x_1^{(1)},p_1), y_1^{(2)} = f(x_1^{(2)},p_1), ..., y_1^{(n_1)} = f(x_1^{(n_1)},p_1)$$

$$y_2^{(1)} = f(x_2^{(1)},p_2), y_2^{(2)} = f(x_2^{(2)},p_2), ..., y_1^{(n_2)} = f(x_1^{(n_2)},p_2)$$

$$...$$

$$y_i^{(1)} = f(x_i^{(1)},p_i), y_i^{(2)} = f(x_i^{(2)},p_i), ..., y_1^{(n_i)} = f(x_1^{(n_i)},p_i)$$

$$...$$

If given a set of $(x_i^{(1)},y_i^{(1)}), (x_i^{(2)},y_i^{(2)}), ...$, we can estimate the value of $p_i$ by means of optimization such as gradient descent or any other algorithms.

My question is : Can the parameter estimation problem described above, which is conventionally solved by optimization methods, be solved by deep learning?

Since $f$ is an explicit math formula, a very large training set can be easily generated by sampling $x$ and $p$ from some sort of prior distribution.

1. Since DL itself is an optimization problem (or parameter estimation problem), what I found is mainly basic tutorials of DL rather than something directly relevant.
2. A similar question on Quora: How can I apply machine learning to parameter estimation problem in control theory? Is there any example study you can suggest?. The only answer to this question is very brief and only provides a 132-page thesis. I scanned the thesis and found that it seemed to solve a rather complicated image processing problem by reinforcement learning. I'm not sure if it is relevant.
3. A similar question on StackOverflow: Can I do nonlinear regression with deep learning ? How?. But the author tried to use deep learning to find the relationship between $x$ and $y$, which is different from my question: the function $f$ is already known, I just want the $p$.

The intuition of solving parameter estimation by deep learning instead of direct optimization could be:

1. Optimization methods such as gradient descent is prone to find a local optima. Maybe DL will have some magic.
2. In practice, the parameter estimation problem may need to be solved many times for different set of $(x,y)$, which requires speed. So conventional optimization methods may be too slow for this. In a real case that prompts me to ask this question, each parameter estimation takes 0.6s by the lsqnonlin function of MATLAB. In contrast, although training a neuron network is time-consuming, getting the output from the input only takes several milliseconds for a pre-trained network.
3. The formula $y=f(x,p)$ may be used in practice for many years and proves to be very effective. The parameter $p$ also has clear physical meanings and can indicate the actual conditions of the system/machine/experiment. Both $x$ and $y$ can be easily obtained. So I want to reserve the formula rather than use a blackbox-like network to predict $y$ from $x$ as most DLs do.

The difficulties I met when solving the problem myself:

1. Suppose the size of the $(x_i,y_i)$ set is $n_i$: $(x_i^{(1)},y_i^{(1)}), (x_i^{(2)},y_i^{(2)}), ..., (x_i^{(n_i)},y_i^{(n_i)})$. The problem is: $n_i$ is not fixed. If $n_i=5$, we may estimate $p_i$ with a great error. And if $n_i=1e6$, we may get a very precise estimation of $p_i$. But in both case we can get an estimation of $p_i$ at last. However, multi-layer perceptron (MLP) or convolution neuron network (CNN) both accept fixed number of inputs. If you do image processing, you even need to resize every image to the same size first. It seems that MLP and CNN don't suit for this question.
2. The $(x,y)$ set has no order as time-series. The 1 and 2 in $(x_i^{(1)},y_i^{(1)})$ and $(x_i^{(2)},y_i^{(2}))$ are just indices, which don't imply that $(x_i^{(1)},y_i^{(1)})$ is necessarily before $(x_i^{(2)},y_i^{(2}))$. So recurrent neuron network (RNN) doesn't seem to be a natural option.

No. Deep learning is not useful or relevant here. Deep learning is for classification. You don't have a classification task. Rather, you have an optimization task: find the parameter value $p$ that causes the best match between the given $x$-values and $y$-values.

Deep learning isn't magic sauce that you can apply to anything you might ever think of and expect it to solve your problem. Your question is an instance of the XY problem. There may be suitable methods to speed up optimization, but you'll probably need to ask a new question and tell us something about the structure/properties of $f$ and the typical range of values for $n_i$.

• Thank you for your answer. It seems that I have made a canonical mistake :). I may not be an expert in machine learning, but as far as I know, the fitnet function of matlab implements function fitting using a neuron network. Although the network in fitnet is not deep, it may indicate that neuron network is capable of something beyond classification of finite classess. This is one of my puzzles about neuron network: I do find classification only in frameworks like tensorflow, but also come across fitting in more traditional tools like the fitnet function. – Wei Feng May 9 '17 at 3:04
• While I agree that the problem as stated doesn't make sense, there is no reason that a deep learning model can't be applied to a continuous labels (i.e., regression). – Nicholas Mancuso May 9 '17 at 3:30
• What if $n_i$ is fixed, say $n_i=50$? Does it become a common regression problem? – Wei Feng May 9 '17 at 10:01
• @WeiFeng, Yeah, I'm aware you can use deep learning for regression. I don't know whether one would call your problem a regression problem, but in any case, it's not the kind of problem solved by deep learning. Whether $n_i$ is fixed doesn't change whether it can be viewed as a regression problem. If you have follow-up questions, use 'Ask Question' rather than posing them in a comment. This is a question-and-answer site, so we stick to a narrow format: one question; one answer. If you'd like a solution to the problem I suggest you follow the advice in the last sentence of my answer. – D.W. May 9 '17 at 13:30
• This problem is stated as a vector regression problem (with a fixed f as an approximator). You are estimting parameters for a set of inputs mapping via f to a set of outputs . Clearly the objective you wish to minimise is something like empirical risk minimisation. You state in your answer that the problem is find a best p such that y = f(x,p) , which is min_p || y - f(x,p) ||^2 . If f was not a fixed formula then you can use a NN to estimate f. I do agree given f is fixed NNs are not suitable. This case of optimisation equates to regresion constrained to a specific RK-hilbert-space. – Francisco Vargas Apr 11 '18 at 13:20

I am interested in this question as well. And I did some research. It turns out that Tensorflow is able to do the single function optimization using SGD, Adam, etc.

Here is a very detailed tutorial.

http://patwie.com/2017/01/04/optimization_with_tf.html

Since the Nonlinear Regression is kind of minimization problem such as minimizing the sum of squared residual, I guess we are able to feed a single objective function of unknown parameters and all the data into Tensorflow to solve the problem.

I have not tried it yet. I will come back with new results.

However, it will not be deep learning or Neural Network anymore.

It just uses the Tensorflow's power to facilitate the optimization.

update:

I did run a model with nonlinear objective function and it works. And then I am thinking that it is a Neural Network with a nonlinear hidden layer.

1st input layer/data/x -> 2nd hidden layer/nonlinear f(x, theta) -> 3rd output layer yhat

the model is

$$\hat y = f(x, \theta)$$

not a linear model

$$a = wx + b$$

The objective function is the sum of squared error: $\sum[(y- \hat y)^2]$.

The Tensorflow use the forward and backward propagation to solve the problem.

We estimate the $\theta$s instead of $w \quad \text{and} \quad b$.

I will MXnet and other packages to see if the idea works as well.

• Welcome to Computer Science! It is good that you are also interested in the question, but it seems that your post doesn't answer the question. SGD is linear only and you only guess that the single function could be fed to Tensorflow. Even if it supports non-linear minimization, the link connecting it to the problem is missing. If you want to get back after some research but it will be neither deep learning nor Neural Network it will also miss the main point of the question as stated. – Evil Oct 27 '17 at 22:13
• Thanks. Evil. I am aware this problem. However, I am thinking that it is a one layer Neural Network. And in solving this problem, we still use the forward and backward propagation. I did try to run a nonlinear optimization problem in Tensorflow and it works. – Jon Duan Oct 31 '17 at 5:46
• your link for the tutorial is broken – Xitcod13 Apr 25 '18 at 20:06