# Combining multiple HMM models

Is there any way to combine multiple Hidden Markov Models trained from different sets of data? For example, I want to detect the phases of a sequential activity. I collect two sets of data by using two types of sensors: accelerometer, and video stream. I create two HMM models from the accelerometer and video stream, respectively. To increate accuracy, can I combine the two models? Yes. Construct the product automaton.

The "product construction" is a standard construction from automata theory class (e.g., typically taught in the context of finite-state automata). It can be applied to HMM's as well, and it will do exactly what you want.

For your example, the resulting HMM will have 9 states. Each state in the resulting HMM will a pair $(s,t)$, where $s$ is a state of the accelerometer HMM and $t$ is a state of the video stream HMM. Transitions between them and probabilities are derived in the obvious way. For instance, if there is a possible transition $s \to s'$ in the accelerometer HMM and a possible transition $t \to t'$ in the video stream HMM, then the product HMM will have a transition $(s,t) \to (s',t')$ (and its probability will be the product of the probabilities of the underlying transitions). The output distribution is obtained by multiplying probabilities: if $p(x|s)$ denotes the probability of outputting $x$ when in state $s$ in the accelerometer HMM, and $p(x|t)$ the probability of outputting $x$ when in state $t$ in the video stream HMM, then we use $p(x|(s,t)) = p(x|s) p(x|t)$ as the probability of outputting $x$ when in state $(s,t)$ in the product HMM.

This assumes that the internal states of the two HMM processes are independent. That's something you'll need to verify yourself, to see whether the independence assumption seems reasonable in your particular application domain.

I would just use product states/transitions and probabilities.

Say accelerometer has states A, B...

Say video steam has states 1, 2...

Then we denote probability of transition from state 'foo' to state 'bar' while emitting character baz as P(foo,bar,baz).

Introduce a new HMM with states A1, A2, B1 etc. and then combine outputs from accelerometer and video stream into a product space of events such than an output in the new HMM is a pair (!,@) where ! is an event in accelerometer and @ is an event in video stream.

Then P(A1,B2,(!,@)) = P(A,B,!) * P(1,2,@).

I would then continue training it to capture any lack of independence however.