Assume that we have access to a generative model $G$, that takes as input a state-action pair $(s,a)$ and outputs a sample consisting of a successor state-observation pair $(s',o)$, that is, $(s',o)\sim G(s,a)$.
It can be assumed that the states, actions, and observations all take on a finite number of values (can be positive integers).
Is it possible to use the generative model to estimate the probability $P(o|s,s',a)$?
- Generate $(s',o)\sim G(s_0,a_0)$
- If $s'=s_0'$, add a particle to bin $b(o)$
- Return to step 1
Continue the above three steps for $M$ runs so that we have $M$ particles in different bins (each bin corresponding to an observation we have seen so far). Now how do we convert the set of particles to probabilities?