So on my course we're dealing with a more basic, simple varition of the Bellman Equation.

V(S) = max a(R(s,a)+yV(s'))

As far as I can tell...

V(S) = Value of a state

max a = Maximum over all the actions

R(s,a) = Reward of a state and action

y = Gamma (Discounting factor)

V(s') = The end state

So how I've always understood equations is by translating them into simple English so please indulge me.

Here's what I think this equation means and I'd like someone to tell me if I'm right or wrong and if I'm wrong rephrase the correct perspective in plain in English as well if possible.

V(S) = max a(R(s,a)+yV(s'))

"The value of a state is equal to the maximum over all the actions, multiplied by the reward of a state and action, added to the discounting factor multiplied by the end state."

Is that correct?

Additional Questions:

(1.) Can "V(s)" be any state or is it the state closest to the end state?

(2.) Presuming you found "V(s)" for the state right before the end state and you wanted to find the value of the previous state before the one you just found. Can you reuse the the equation and make the value of the state you just found the new end state or "V(s')"?

(3.) What does the equation mean by the maximum of all the actions? Does it mean you get all the values of the actions and add them up or instead does it mean do you only take the highest value of all the actions and use that one exclusively?

  • 2
    $\begingroup$ Welcome to Computer Science! Note that you can use LaTeX here to typeset mathematics in a more readable way. See here for a short introduction. $\endgroup$ – Raphael Jul 10 '18 at 16:41
  • 3
    $\begingroup$ "So how I've always understood equations is by translating them into simple English" -- that approach has limits, and you might just have reached them. I recommend getting used to using mathematics as a language if you plan to go any deeper into the rabbit hole. $\endgroup$ – Raphael Jul 10 '18 at 16:42
  • 1
    $\begingroup$ You may want to check out some older questions about dynamic programming. You are not alone in not quite grokking what happens in DP, and it has been explained many times already. FWIW, the first thing you'll have to understand is recursion (I mention this because you don't). $\endgroup$ – Raphael Jul 10 '18 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.