I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can implement value iteration using a function approximator. I couldn't find any implementation example online. Can you please point me some references?
There are two primary methods to deal with continuous state MDPs. 1. State-space discretization. 2. Value function approximation. As for value function approximation, you can either go for a deterministic/stochastic black-box model or opt for fitted value iteration. You can find the algorithm for the same in pages 10-13 of this link.