6 votes
Accepted

Why do we use the log in gradient-based reinforcement algorithms?

We often take the logarithm because: Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of ...
user avatar
  • 143k
6 votes

The meaning of discount factor on reinforcement learning

The discount factor does not represent the likelihood to reach the state $s′ $from the state $s$. That would be $p(s'|s,a)$, which is not used in Q-Learning, since it is model-free (only model-based ...
user avatar
  • 481
5 votes
Accepted

What does it mean to have a continuous action space w.r.t. to reinforcement learning?

When you're driving of your car and you turn the wheel, is that a discrete or a continuous action? It's continuous, because you can control how much you turn the wheel. How much do you press the gas ...
user avatar
  • 143k
4 votes
Accepted

What's the difference between Adaptive Control and Hierarchical Reinforcement Learning?

The difference between the two tasks really comes down to the level of continuity assumed in the models of the problem. In adaptive control, continuity is assumed at all levels; the problem space and ...
user avatar
  • 641
4 votes

Why does ε-greedy $Q$-learning not oscillate?

The assumptions for Q-learning's convergence proof require that: [...] all state-action pairs be visited infinitely often So the convergence is understood in a mathematical sense - the Q-function ...
user avatar
  • 598
4 votes
Accepted

What is the difference between apprenticeship and imitation learning?

In general, yes, they are the same thing, which means to learn from demonstration (LfD). But, usually, apprenticeship learning is mentioned in the context of "Apprenticeship learning via inverse ...
user avatar
  • 481
3 votes
Accepted

Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The question, as written, certainly suggests that it's a "good policy" to bet a=50 for s=50 , while not "betting it all" with a=49 for s=51. However, in an e-mail discussion surrounding this (here), ...
user avatar
  • 146
3 votes

Some questions on kernels and Reinforcement Learning

Is that valid? My question is how can I distinguish between a valid kernel and a nonvalid kernel if I'm given a kernel in the test? Yes, it will be a valid kernel for all practical purposes, because ...
user avatar
  • 640
3 votes

Reinforcement learning without self-play

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to ...
user avatar
  • 31
3 votes

Is TD-learning considered a model-based algorithm?

The previous answer is wrong. TD-learning is a model-free algorithm. You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration. Source: https:/...
user avatar
3 votes

Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?

I just found out the answer and it's actually pretty simple: while there is a good linear policy for the mountain car task, the value function itself is non-linear. The state space of this task is ...
user avatar
  • 481
2 votes
Accepted

Reference request: Introduction to reinforcement learning with hand calculation examples

Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents, fully available online, has one such example for Q-learning. Sutton & Barto's Reinforcement Learning: An ...
user avatar
2 votes
Accepted

When do the gradients exist in a Reinforcement Learning setting?

The gradient doesn't exist / isn't well-defined for non-differentiable functions. What they mean by that statement is that there is an analogous version of gradients that can be used, instead of the ...
user avatar
  • 143k
2 votes

Dyna-Q in non-deterministic domains

One approach is outlined in Tucker Balch's Machine Learning for Trading course on Udacity: Learning T Learning R In a nutshell: create a $T_c$ table ("T count") that counts the number of times each ...
user avatar
  • 121
2 votes
Accepted

Can I have actions that, if taken, cannot be taken again in the following N timesteps?

This is not a limitation. Which actions are valid can depend on the current state of the system. To enforce your requirement (changing speed at time T means you can't change again until time T+N), ...
user avatar
  • 143k
2 votes
Accepted

Does an RL agent need to experience a complete episode in order to learn?

If the agent gets no feedback whatsoever and has no information about the structure of the search space (just blind search, hoping to get lucky), then no, it can't learn. It has nothing to learn from....
user avatar
  • 143k
2 votes

Exploration in reinforcement learning when state space is huge

In the Atari paper, an $\epsilon$-greedy strategy is used for state-space exploration. This means that the algorithm makes the deep network learn a greedy strategy to pick an action that maximises the ...
user avatar
2 votes
Accepted

Approximate value iteration for continuous state space MDPs

There are two primary methods to deal with continuous state MDPs. 1. State-space discretization. 2. Value function approximation. As for value function approximation, you can either go for a ...
user avatar
2 votes

What are the basics of CS i should know,before I start my journey into machine learning

Computer science is a very broad subject area, and many of its sub-disciplines have little or no overlap with others. For example, knowing the basics of operating systems design, compiler design or ...
user avatar
  • 1,549
2 votes
Accepted

Reinforcement learining basics - is it possible to deal with environment with random events

Yes, in general reinforcement learning algorithms can handle random events in the environment. Think about the q-learning update rule $$ Q(s_t,a_t)= Q_{s_t, a_t} + \alpha \left(r_{t+1} + \gamma max_aQ(...
user avatar
  • 354
1 vote

How can I improve this RL algorithm?

There's no reason to think that reinforcement learning would be effective here. Indeed, the surprise would be if it did work! Machine learning isn't a magic silver bullet. I wouldn't expect it to ...
user avatar
  • 143k
1 vote

Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The best policy, known as bold play and figured out by Dubbins and Savage, is to always gamble as much as possible (but not more than is necessary to reach the goal). See for example an exposition by ...
user avatar
1 vote

Reinforcement learning without self-play

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a ...
user avatar
1 vote
Accepted

Reinforcement Learning - Q Learning

At t=1, you just took action a = d at state s = S and ended in ...
user avatar
  • 481
1 vote

Is TD-learning considered a model-based algorithm?

For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo. As for control problems, Q-learning & Sarsa are both TD algos, and it'...
user avatar
1 vote

Is TD-learning considered a model-based algorithm?

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is ...
user avatar
  • 11
1 vote
Accepted

Difference between SNN RL and DNN RL?

Daniel Rasmussen replied via email: SNNs take less long to train This really depends on the training method, and the implementation. If you just implemented an abstract DNN approach ...
user avatar
  • 641
1 vote
Accepted

Using a combination of spatial and non-spatial inputs for convolutional neural networks

Yes. One way is to start with several convolutional layers, and then end with one or two fully connected layers. You can add the other features as inputs to the first of those fully connected layers....
user avatar
  • 143k
1 vote

How is the environment designed for testing a reinforcement learning algorithm?

In reinforcement learning, the "environment" is typically a set of states the "agent" is attempting to influence via its choice of "actions". For example, in "Reinforcement learning design for ...
user avatar
  • 1,582
1 vote

single agent vs multiple agent reinforcement learning

A key difference between RL and MARL arises when you consider that other agents are strategic and their behaviour is adaptive. Game theoretical concepts are very important within the scope of MARL and ...
user avatar
  • 315

Only top scored, non community-wiki answers of a minimum length are eligible