# Questions tagged [markov-chains]

The tag has no usage guidance.

74 questions
Filter by
Sorted by
Tagged with
13 views

### Is a metaheuristics a Markov Decision Process?

Is a metaheuristics optimization algorithm, such as simulated annealing and genetic algorithms, a Markov Decision Process?
• 547
20 views

### Why is data generated from a randomly constructed Markov chain not compressible?

For a project I'm working on, I need to generate some data to test the Gzip compressor, and I wrote my own implementation of Markov chain for it, and it's a 2-level nested one. But when I generate ...
• 161
24 views

• 636
94 views

### Solving the following recurrence relation derived from a Markov chain

I have the following system of recursive relations on $y_{i,j}$ that are derived from a Markov chain and that I am having difficulty in solving. For $i\ge 1$ and $j \ge 1$, we have y_{i,j} \times (...
1k views

### Absorbing Markov Chains: An efficient algorithmic approach

Following this procedure I have successfully written a program to calculate the probability of ending in a given absorbing state given the initial state. The procedure is as follows: Given the ...
• 177
108 views

### Form of conditional observation probabilities in a POMDP

Consider a partially observable Markov decision process (POMDP), see here for a complete definition. My question is in relation to the conditional observation probabilities (denoted by $O(o|s',a)$ in ...
• 371
1 vote
213 views

### Calculate expected values of "Craps Game" with the help of Prism Model Checker

I have modelled the craps game (https://en.wikipedia.org/wiki/Craps#Rules_of_play) as a dtmc with the prism model checker: ...
202 views

### probability matrix from digraph adjacency matrix

All I have in hand is a adjacency matrix of a digraph with equal weight on every edge, is there a very simple way to convert this to a state change probability matrix?
363 views

### Convergence of Markov model

I was learning Hidden Markov model, and encountered this theory about convergence of Markov model. For example, consider a weather model, where on a first-day probability of weather being sunny was 0....
The book Cycle Representations of Markov Processes solves the problem of Mapping Stochastic Matrices induced from a Markov Chain into Partitions using a $\lambda$-preserving ($\lambda$ is a Lebesgue ...