I got a question about who and how sets the transation function values in markov decision processes?
I mean when some says that an agent, in real world grid, is going to step up by %80 and left/right by %10 each. How did he know that? Do we have to determine the transition probability for each step or agents have to? Simply, it should be non-deterministic process, so how could someone tell the probability of that transition?
It should be %25 for each direction!!
I mean when some says that an agent, in real world grid, is going to step up by %80 and left/right by %10 each. How did he know that?
There are a couple of scenarios that could lead to an agent stepping up with probability 0.8 and left/right with probability 0.1/0.1. The simplest case, and the one you would likely encounter first is that the agent takes an action determined by the agent's policy but the environment dynamics do not map that state-action pair deterministically to a next state, but to a distribution over possible next states. Then taking the same action in the same state multiple times will not always result in the same transition. The agent may choose to go up, but the environment dynamics assign a small probability that if the agent moves up while in the current state the agent may end up moving left or right instead. The dynamics are a feature of the environment, they are independent of the agent, for example they may be observed probabilities in a video game which was developed by other people than the ones develop the RL algorithm to play it.
Another scenario that could explain the probabilities you asked about is the agent could learn a stochastic policy, meaning instead of using a deterministic policy $$\pi:S\to A$$ The agent uses a stochastic policy $$\pi:S\to prob(A)$$ which maps states to probability distributions over actions. In this case, even if the environment is deterministic the agent may choose to move up with probability 0.8 and left or right with probability 0.1 each.
It should be %25 for each direction!!
This could happen if the environment dynamics did not allow the agent to make a meaningful choice at that state, or the agent's policy assigned each action an equal probability at that state. RL agent's are often explicitly programmed to behave highly randomly in the beginning, maybe even following a uniform random policy, which would assign a probability of .25 for each direction in each state. The amount of random actions the agent takes is then annealed over time as more information is obtained about the environment and the estimate of the value function is refined.
