7 votes
Accepted

Why do we use the log in gradient-based reinforcement algorithms?

We often take the logarithm because: Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of ...
D.W.'s user avatar
  • 159k
5 votes
Accepted

What does it mean to have a continuous action space w.r.t. to reinforcement learning?

When you're driving of your car and you turn the wheel, is that a discrete or a continuous action? It's continuous, because you can control how much you turn the wheel. How much do you press the gas ...
D.W.'s user avatar
  • 159k
4 votes
Accepted

What is the difference between apprenticeship and imitation learning?

In general, yes, they are the same thing, which means to learn from demonstration (LfD). But, usually, apprenticeship learning is mentioned in the context of "Apprenticeship learning via inverse ...
rcpinto's user avatar
  • 471
3 votes
Accepted

Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The question, as written, certainly suggests that it's a "good policy" to bet a=50 for s=50 , while not "betting it all" with a=49 for s=51. However, in an e-mail discussion surrounding this (here), ...
shaneb's user avatar
  • 146
3 votes

Some questions on kernels and Reinforcement Learning

Is that valid? My question is how can I distinguish between a valid kernel and a nonvalid kernel if I'm given a kernel in the test? Yes, it will be a valid kernel for all practical purposes, because ...
Maxim's user avatar
  • 640
3 votes

Reinforcement learning without self-play

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to ...
fezzik's user avatar
  • 31
3 votes

Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?

I just found out the answer and it's actually pretty simple: while there is a good linear policy for the mountain car task, the value function itself is non-linear. The state space of this task is ...
rcpinto's user avatar
  • 471
3 votes

Is TD-learning considered a model-based algorithm?

The previous answer is wrong. TD-learning is a model-free algorithm. You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration. Source: https:/...
erensezener's user avatar
3 votes
Accepted

How to setup the Bellman Equation as a linear system of equation

I found the full answer in a video by David Silver. The idea is easy enough. The underlying matrix equation is $$ v = R^\pi + \gamma P^\pi_{s, s'} v $$ Where $v$ is the value function, $R^\pi$ is the ...
krishnab's user avatar
  • 171
2 votes

single agent vs multiple agent reinforcement learning

A key difference between RL and MARL arises when you consider that other agents are strategic and their behaviour is adaptive. Game theoretical concepts are very important within the scope of MARL and ...
Juan Leni's user avatar
  • 325
2 votes

Is TD-learning considered a model-based algorithm?

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is ...
pmorais's user avatar
  • 21
2 votes
Accepted

Reference request: Introduction to reinforcement learning with hand calculation examples

Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents, fully available online, has one such example for Q-learning. Sutton & Barto's Reinforcement Learning: An ...
Sean Easter's user avatar
2 votes
Accepted

When do the gradients exist in a Reinforcement Learning setting?

The gradient doesn't exist / isn't well-defined for non-differentiable functions. What they mean by that statement is that there is an analogous version of gradients that can be used, instead of the ...
D.W.'s user avatar
  • 159k
2 votes

Dyna-Q in non-deterministic domains

One approach is outlined in Tucker Balch's Machine Learning for Trading course on Udacity: Learning T Learning R In a nutshell: create a $T_c$ table ("T count") that counts the number of times each ...
Michael's user avatar
  • 121
2 votes
Accepted

Can I have actions that, if taken, cannot be taken again in the following N timesteps?

This is not a limitation. Which actions are valid can depend on the current state of the system. To enforce your requirement (changing speed at time T means you can't change again until time T+N), ...
D.W.'s user avatar
  • 159k
2 votes
Accepted

Does an RL agent need to experience a complete episode in order to learn?

If the agent gets no feedback whatsoever and has no information about the structure of the search space (just blind search, hoping to get lucky), then no, it can't learn. It has nothing to learn from....
D.W.'s user avatar
  • 159k
2 votes
Accepted

Approximate value iteration for continuous state space MDPs

There are two primary methods to deal with continuous state MDPs. 1. State-space discretization. 2. Value function approximation. As for value function approximation, you can either go for a ...
RICHA VERMA's user avatar
2 votes

Exploration in reinforcement learning when state space is huge

In the Atari paper, an $\epsilon$-greedy strategy is used for state-space exploration. This means that the algorithm makes the deep network learn a greedy strategy to pick an action that maximises the ...
RICHA VERMA's user avatar
2 votes

What are the basics of CS i should know,before I start my journey into machine learning

Computer science is a very broad subject area, and many of its sub-disciplines have little or no overlap with others. For example, knowing the basics of operating systems design, compiler design or ...
gandalf61's user avatar
  • 1,589
2 votes
Accepted

Reinforcement learining basics - is it possible to deal with environment with random events

Yes, in general reinforcement learning algorithms can handle random events in the environment. Think about the q-learning update rule $$ Q(s_t,a_t)= Q_{s_t, a_t} + \alpha \left(r_{t+1} + \gamma max_aQ(...
awillia91's user avatar
  • 394
1 vote

Is deep learning appropriate to approximate dynamic programming problems?

A good start would be to understand what reinforcement learning is; and identify which problems are reinforcement learning problems and which aren't. Deep reinforcement learning is only relevant if ...
D.W.'s user avatar
  • 159k
1 vote
Accepted

Existence of Optimal Policy for infinite-state MDPs

If the state set is Polish (e.g the set of all real numbers), finiteness of action spaces suffices to conclude that the MDP always has an optimal stationary policy - refer to "Markov Decision ...
Toghs's user avatar
  • 36
1 vote

Eligibility trace and the role of gamma and lambda

At least for gamma I can answer it generally without looking in the book: Gamma is commonly used as the discount factor. That is a number in (0, 1) which defines how much of the reward you got in ...
Martin Thoma's user avatar
  • 2,360
1 vote

Inverse Reinforcement Learning and Apprenticeship Learning differences

Apprenticeship Learning uses Inverse Reinforcement Learning and there is a subtle difference between the two. Inverse Reinforcement Learning tries to recover the reward function ($R^*$) from ...
RICHA VERMA's user avatar
1 vote

How can I improve this RL algorithm?

There's no reason to think that reinforcement learning would be effective here. Indeed, the surprise would be if it did work! Machine learning isn't a magic silver bullet. I wouldn't expect it to ...
D.W.'s user avatar
  • 159k
1 vote

Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The best policy, known as bold play and figured out by Dubbins and Savage, is to always gamble as much as possible (but not more than is necessary to reach the goal). See for example an exposition by ...
Yuval Filmus's user avatar
1 vote

Reinforcement learning without self-play

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a ...
Sridhar Thiagarajan's user avatar
1 vote

Is TD-learning considered a model-based algorithm?

For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo. As for control problems, Q-learning & Sarsa are both TD algos, and it'...
T. Jiang's user avatar
1 vote
Accepted

Difference between SNN RL and DNN RL?

Daniel Rasmussen replied via email: SNNs take less long to train This really depends on the training method, and the implementation. If you just implemented an abstract DNN approach ...
Seanny123's user avatar
  • 641
1 vote
Accepted

Using a combination of spatial and non-spatial inputs for convolutional neural networks

Yes. One way is to start with several convolutional layers, and then end with one or two fully connected layers. You can add the other features as inputs to the first of those fully connected layers....
D.W.'s user avatar
  • 159k

Only top scored, non community-wiki answers of a minimum length are eligible