Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order to select the best action at state $s$, the resulting state $s'$ for every possible action must be computed, so we can get $V(s')$. Thus, we need a model for computing $s'$ from a $(s,a)$ pair. Is that correct? Does that mean that TD-learning can be considered a model-based technique? Or in the case of TD-learning we consider it just a value updating algorithm and ignore the control part, thus removing the need for a model?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Perhaps I'm missing some key aspect of the motivation behind this question, but: Does it matter what we consider it? A rose by any other name would smell as sweet. $\endgroup$– D.W. ♦Commented Jun 6, 2016 at 22:10
-
$\begingroup$ Yes, it does when you write a thesis proposal and must refer correctly to the described algorithms, specially if you're drawing clear lines between them. $\endgroup$– rcpintoCommented Jun 6, 2016 at 23:40
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
The previous answer is wrong. TD-learning is a model-free algorithm.
You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration.
Source: https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node60.html
$\begingroup$
$\endgroup$
1
Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is useless in terms of decision making if you don't also have a model of the MDP. The reason Q-learning and SARSA are considered model-free is because you don't need such a model of the MDP in order to extract a policy from a Q function.
-
$\begingroup$ I think that there is a misunderstanding here. The word
modelis used in two different contexts. @erensezener is correct. $\endgroup$ Commented Dec 19, 2016 at 18:16
$\begingroup$
$\endgroup$
For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo.
As for control problems, Q-learning & Sarsa are both TD algos, and it's obivious that they are both model-free control algos.
However, if you want to use $V(s)$ in a control problem -- it's definitely model-based and that is exactly why we need a state-action function. (You might turn to David Silver's video. In the Lecture 6 (around 51:20), he said "... we need to use action value function Q, so we can make it model-free and still be able to pick the greedy action." )
In short, TD is a really big class of algos, and you cannot simply say "TD is model-free" nor "TD is model-based".
Since MC is generally considered model-free but MCTS is also model-based.
