Machine Learning vs System Identification?

Question

Could anyone explain to me the differences & similarities between machine learning and system identifications? Are these just two names of the same thing? In this page, they say:

Machine learning and system identification communities are faced with similar problems where one needs to construct a model from limited or noisy observations.

I've also read the early chapters of the famous book Pattern Recognition and Machine Learning by Christopher M. Bishop. So far, my conclusion is that the problem that system identification is trying to solve is a subset of what machine learning is trying to solve.

Answer 1

System identification is the science of constructing dynamical models from observed data. There are two main approaches: Prediction Error Identification (PEI) and Subspace Identification (SID). Both of them are delivering a so called parametric model, that is to say, a model of a fixed structure. Usually it is the case that the user selects the structure of the underlying system (especially in the PEI methods) or at least the order of the system (in both methods). Even though it is not necessary, a low order system is sought (that is to say, the number of the basis coefficients is relatively small) because it is often used for control purposes, so we have to keep it as simple as possible to avoid computational issues etc. This model can be used to make predictions about the future behaviour of the system given some inputs.

On the other hand, machine learning (ML) has two main branches, classification and regression algorithms. The latter ones are also used for prediction purposes. Two of the most famous approaches in machine learning are Support Vector Machines (SVM) and Gaussian processes (GP). The main difference with the system identification techniques is that the ML techniques are delivering a non-parametric model. The latter means that the prediction for a new input is given as a function of the data points used for the "training" (learning, identification) of the model. Therefore, if we used N=1000 data points for the training, then the prediction would be expressed as a function of these data points. ML methods are more flexible since they don't require any structure selection from the user, but they face other limitations (e.g. computational effort).

Until recently the ML and the system identification techniques were developing independently. But in the latter years there is a great effort to establish a common ground (e.g. see the paper "Four encounters with system identification" from Ljung)

Answer 2

Context: SysID and controls guy who got into ML.

I think user110686's answer does a fair job of explaining some differences. SysID is necessarily about dynamic models from input/output data, whereas ML covers a wider class of problems. But the biggest difference I see is to do with (a) memory (number of parameters); (b) end use of the "learned" model . System Identification is very much a signal processing approach considering frequency domain representations, time-frequency analysis etc. Some ML folks call this "feature engineering".

(a) Memory: SysID became prominent long before ML as a research field took shape. Hence statistics and signal processing were the primary basis for the theoretical foundations, and computation was scare. Hence, people worked with very simple class of models (Bias-Variance tradeoff) with very few parameters. We are talking at most 30-40 parameters and mostly linear models even for cases where people clearly know the the problem is non-linear. However, now computation is very cheap but SysID hasn't come out of its shell yet. People should start realizing that we have much better sensors now, can easily estimate 1000s of parameters with very rich model sets. Some researchers have attempted to use neural networks for SysID but many seem reluctant to accept these as "mainstream" since there aren't many theoretical guarantees. For anything not linear, its going to be hard getting guarantees anyway, so I am curious as to how the field will proceed.

(b) End use of learned model: Now this is one thing SysID got very correct, but many ML algorithms fail to capture. It is important to recognize that for the target applications, you are necessarily building models that can be used effectively for online optimization. These models will be used to propagate any control decisions made, and when setting this up as an optimal control problem, the models become constraints. So when using an extremely complicated model structure, it makes the online optimization that much more difficult. Also note that these online decisions are made in the scale of seconds or less. An alternative proposed is to directly learn value function in an off-policy manner for optimal control. This is basically reinforcement learning, and I think there is good synergy between SysID and RL.

Answer 3

I'd like to add that there are also non-parametric approaches to system identification. See MATLAB's SysId toolbox or Ljung's book for details. Non-parametric approaches are often used to first identify the class of models for later parametric studies. Also, it is important to separate the estimation problem from the control problem (think OODA loop). In identifying a system it is often the goal simply to characterize the system without any specific control inputs of the type that will be designed later (but this is not always possible). Finally, I think it is helpful to realize that a system is from a mathematical point of view an operator that maps a function space to a function space. Hence differential equations are often the kinds of things identified and these map functions to functions. The functions in SysID are often continuous functions of time, aka continuous-time signals. (But they can be discrete-time as well.) Thus, SysID is not just attempting to map real numbers (or vectors) to real numbers (or vectors); it seeks to identify the best operator (LTI, LTV, non-linear, etc.) that maps input signals to output signals.

Answer 4

This is murky indeed. I can't say I agree with the distinctions mentioned about parameterization. I think there are parametric and non-parametric examples of both. My 2 cents: a major distinction is the direction of causality in the system. In ML the "real-world" direction of causality can go in either direction x-to-y or y-to-x, but the model always has y (the "target") at the output and x (the "observation") at the input. In sys ID, x tends to drive the system, thus generating y.

Answer 5

Machine Learning: modeling for static model and dynamic model , System Identification: focus on dynamic model or dynamic process