Learning dynamics

Question

I recently learned about Q-learning, a reinforcement learning technique that directly estimates the expected value of taking an action in a state.

I'm wondering if there exists techniques to do "dynamics learning", in order to estimate the dynamics of a system. A "dynamics learning" agent might choose actions which help it estimate the state transition function, or to estimate parameters of some known transition function.

For example, a "dynamics learning" agent in the cart-pole system would discover a function that approximates the equations of motion of the cart-pole. Or, the agent might know these equations, but not parameters of the system, like the inertial moment of the pendulum or the mass of the cart.

What techniques are there to do "dynamics learning"?

Answer 1

There are three issues. First, you must choose a class of models for the dynamics. Second, you must construct a training set, by taking the agent down different trajectories to explore the state space. Third, you need a way to learn/infer a particular dynamics model from this training set. There are different ways to instantiate each of these tasks.

In robotics, one common choice is to use a Markov decision process (MDP) for the dynamics model. This is a convenience choice, because there are relatively standard ways to learn a MDP from a training set, and because given a MDP dynamics model it is well-studied how to construct a controller for the system that takes the dynamics into account. Another option is to use a higher-order Markov decision process or a partially observable Markov decision process, but those can be much more unwieldy to work with.

There are many ways to explore the state space. Chapter 3.1 of Pieter Abbeel's PhD thesis has a nice overview as of 2008.

To infer/learn a MDP given a training set, you can use maximum likelihood estimation. More sophisticated techniques are also possible; see Abbeel's PhD thesis for an overview this as well (Chapters 3 and 4).

More generally, see the literature on system identification from the field of control theory.