# Machine Learning vs System Identification?

Could anyone explain to me the differences & similarities between machine learning and system identifications? Are these just two names of the same thing? In this page, they say:

Machine learning and system identification communities are faced with similar problems where one needs to construct a model from limited or noisy observations.

I've also read the early chapters of the famous book Pattern Recognition and Machine Learning by Christopher M. Bishop. So far, my conclusion is that the problem that system identification is trying to solve is a subset of what machine learning is trying to solve.

-

System identification is the science of constructing dynamical models from observed data. There are two main approaches: Prediction Error Identification (PEI) and Subspace Identification (SID). Both of them are delivering a so called parametric model, that is to say, a model of a fixed structure. Usually it is the case that the user selects the structure of the underlying system (especially in the PEI methods) or at least the order of the system (in both methods). Even though it is not necessary, a low order system is sought (that is to say, the number of the basis coefficients is relatively small) because it is often used for control purposes, so we have to keep it as simple as possible to avoid computational issues etc. This model can be used to make predictions about the future behaviour of the system given some inputs.

On the other hand, machine learning (ML) has two main branches, classification and regression algorithms. The latter ones are also used for prediction purposes. Two of the most famous approaches in machine learning are Support Vector Machines (SVM) and Gaussian processes (GP). The main difference with the system identification techniques is that the ML techniques are delivering a non-parametric model. The latter means that the prediction for a new input is given as a function of the data points used for the "training" (learning, identification) of the model. Therefore, if we used N=1000 data points for the training, then the prediction would be expressed as a function of these data points. ML methods are more flexible since they don't require any structure selection from the user, but they face other limitations (e.g. computational effort).

Until recently the ML and the system identification techniques were developing independently. But in the latter years there is a great effort to establish a common ground (e.g. see the paper "Four encounters with system identification" from Ljung)

-

Context: SysID and controls guy who got into ML.

I think user110686's answer does a fair job of explaining some differences. SysID is necessarily about dynamic models from input/output data, whereas ML covers a wider class of problems. But the biggest difference I see is to do with (a) memory (number of parameters); (b) end use of the "learned" model . System Identification is very much a signal processing approach considering frequency domain representations, time-frequency analysis etc. Some ML folks call this "feature engineering".

(a) Memory: SysID became prominent long before ML as a research field took shape. Hence statistics and signal processing were the primary basis for the theoretical foundations, and computation was scare. Hence, people worked with very simple class of models (Bias-Variance tradeoff) with very few parameters. We are talking at most 30-40 parameters and mostly linear models even for cases where people clearly know the the problem is non-linear. However, now computation is very cheap but SysID hasn't come out of its shell yet. People should start realizing that we have much better sensors now, can easily estimate 1000s of parameters with very rich model sets. Some researchers have attempted to use neural networks for SysID but many seem reluctant to accept these as "mainstream" since there aren't many theoretical guarantees. For anything not linear, its going to be hard getting guarantees anyway, so I am curious as to how the field will proceed.

(b) End use of learned model: Now this is one thing SysID got very correct, but many ML algorithms fail to capture. It is important to recognize that for the target applications, you are necessarily building models that can be used effectively for online optimization. These models will be used to propagate any control decisions made, and when setting this up as an optimal control problem, the models become constraints. So when using an extremely complicated model structure, it makes the online optimization that much more difficult. Also note that these online decisions are made in the scale of seconds or less. An alternative proposed is to directly learn value function in an off-policy manner for optimal control. This is basically reinforcement learning, and I think there is good synergy between SysID and RL.

-
Welcome to Computer Science Stack Exchange, Aravind! – David Richerby Oct 31 '15 at 20:44

Machine Learning: modeling for static model and dynamic model , System Identification: focus on dynamic model or dynamic process

-
You answer is a bit terse, could you elaborate your answer a bit to provide more detail - for example what is the difference (if any - I'm no expert) between the machine learning dynamic modelling and the system identification dynamic modelling - or are you saying that system identification focuses only on dynamic machine learning, whereas the broad area has a static component? (Just ideas for how you could expand your answer to make it better - maybe they're not good ones) – Luke Mathieson Mar 30 '13 at 7:26