# Is TD-learning considered a model-based algorithm?

Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order to select the best action at state $s$, the resulting state $s'$ for every possible action must be computed, so we can get $V(s')$. Thus, we need a model for computing $s'$ from a $(s,a)$ pair. Is that correct? Does that mean that TD-learning can be considered a model-based technique? Or in the case of TD-learning we consider it just a value updating algorithm and ignore the control part, thus removing the need for a model?

• Perhaps I'm missing some key aspect of the motivation behind this question, but: Does it matter what we consider it? A rose by any other name would smell as sweet.
– D.W.
Jun 6 '16 at 22:10
• Yes, it does when you write a thesis proposal and must refer correctly to the described algorithms, specially if you're drawing clear lines between them. Jun 6 '16 at 23:40

The previous answer is wrong. TD-learning is a model-free algorithm.

You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration.

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is useless in terms of decision making if you don't also have a model of the MDP. The reason Q-learning and SARSA are considered model-free is because you don't need such a model of the MDP in order to extract a policy from a Q function.

• I think that there is a misunderstanding here. The word model is used in two different contexts. @erensezener is correct. Dec 19 '16 at 18:16

For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo.

As for control problems, Q-learning & Sarsa are both TD algos, and it's obivious that they are both model-free control algos.

However, if you want to use $V(s)$ in a control problem -- it's definitely model-based and that is exactly why we need a state-action function. (You might turn to David Silver's video. In the Lecture 6 (around 51:20), he said "... we need to use action value function Q, so we can make it model-free and still be able to pick the greedy action." )

In short, TD is a really big class of algos, and you cannot simply say "TD is model-free" nor "TD is model-based".

Since MC is generally considered model-free but MCTS is also model-based.