In reinforcement learning, does policy affect the maximization of the value?

Question

Though the reward was assigned by the environment, the once the policy $\pi$ was fixed, the probability of the action on the states $\pi(a|s)$ could be assigned.

However, this meant given different policy $\pi_i$, the maximization of the value function was not consistent, i.e. the same set of action under different policies $\pi_i$ might lead to different value, i.e. the value function was different. But that's a bit counter intuitive, since the actual reward received by the agent ought to be the same(disregard the damping factor $\gamma$).

In reinforcement learning, does policy affect the maximization of the value? How could there be a maximized value function then?

Answer 1

Indeed in RL the policy $π$ does affect the maximization of the value function. This is a fundamental aspect of RL since the value function is defined w.r.t. a specific policy and different policies lead to different value functions. The optimal value functions correspond to the highest expected return achievable from each state, assuming the agent acts optimally from that state onward.

The maximized value function $V^∗(s)$ is consistent because it represents the maximum expected return obtainable from state $s$ under any policy. This value is unique and defines the upper bound of what is possible from a state. However, the specific policy $π^∗$ that achieves this maximum is not unique, there could be multiple policies that achieve the same optimal value function, but all optimal policies will result in the same optimal value function.