0
$\begingroup$

I have a time-serie and I fit different HMMs on it, each with a different number of hidden states. Now after sampling from the models , I'd like to compare the results with the ground truth data and find the model that gets closer to the real world data in the original time-serie.

For now I simply compared visually the distribution of the values generated by the HMMs and the distribution of values in the time-serie, but I'd like to compute a number indicating which model generates better samples.

$\endgroup$

1 Answer 1

2
$\begingroup$

Compute the likelihood of the observed data, for each model. Then higher the likelihood, the better the fit. The likelihood is just the probability that the model assigns to the observed data, which for HMMs can be computed using dynamic programming.

Be prepared that the more complex the model, the higher the likelihood will be, but that doesn't necessarily mean the model is "better" -- you run the risk of overfitting. Larger HMMs may fit the training data better, but if the HMM is too large, it might perform poorly on new data because of overfitting.

$\endgroup$
2
  • $\begingroup$ I don't know why in all examples I find they train the hmm on the same data they then pass to the model's score function which returns the log likelihood. Shouldn't I just put aside some data (e.g. the last 3 days of observations) on which compute the score? Btw, I already computed the log likelihood and as you said, the better fit is the one with more states, but still I'd like a non visual way to compare the generated samples with the ground truth data $\endgroup$ Apr 30, 2021 at 22:31
  • $\begingroup$ @GerardoZinno, a standard approach would be to use hold-out sets (a validation set and a test set, held out) or cross-validation. For instance, perhaps you could use the validation set to choose the number of states to use. $\endgroup$
    – D.W.
    May 1, 2021 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.