0
$\begingroup$

Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order to select the best action at state $s$, the resulting state $s'$ for every possible action must be computed, so we can get $V(s')$. Thus, we need a model for computing $s'$ from a $(s,a)$ pair. Is that correct? Does that mean that TD-learning can be considered a model-based technique? Or in the case of TD-learning we consider it just a value updating algorithm and ignore the control part, thus removing the need for a model?

$\endgroup$
2
  • $\begingroup$ Perhaps I'm missing some key aspect of the motivation behind this question, but: Does it matter what we consider it? A rose by any other name would smell as sweet. $\endgroup$
    – D.W.
    Jun 6 '16 at 22:10
  • $\begingroup$ Yes, it does when you write a thesis proposal and must refer correctly to the described algorithms, specially if you're drawing clear lines between them. $\endgroup$
    – rcpinto
    Jun 6 '16 at 23:40
3
$\begingroup$

The previous answer is wrong. TD-learning is a model-free algorithm.

You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration.

Source: https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node60.html

$\endgroup$
1
$\begingroup$

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is useless in terms of decision making if you don't also have a model of the MDP. The reason Q-learning and SARSA are considered model-free is because you don't need such a model of the MDP in order to extract a policy from a Q function.

$\endgroup$
1
  • $\begingroup$ I think that there is a misunderstanding here. The word model is used in two different contexts. @erensezener is correct. $\endgroup$
    – Juan Leni
    Dec 19 '16 at 18:16
1
$\begingroup$

For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo.

As for control problems, Q-learning & Sarsa are both TD algos, and it's obivious that they are both model-free control algos.

However, if you want to use $V(s)$ in a control problem -- it's definitely model-based and that is exactly why we need a state-action function. (You might turn to David Silver's video. In the Lecture 6 (around 51:20), he said "... we need to use action value function Q, so we can make it model-free and still be able to pick the greedy action." )


In short, TD is a really big class of algos, and you cannot simply say "TD is model-free" nor "TD is model-based".

Since MC is generally considered model-free but MCTS is also model-based.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.