8
votes
Accepted
Why do we use the log in gradient-based reinforcement algorithms?
We often take the logarithm because:
Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of ...
D.W.♦
- 166k
3
votes
Accepted
Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution
The question, as written, certainly suggests that it's a "good policy" to bet a=50 for s=50 , while not "betting it all" with a=49 for s=51. However, in an e-mail discussion surrounding this (here), ...
- 146
3
votes
Some questions on kernels and Reinforcement Learning
Is that valid? My question is how can I distinguish between a valid kernel and a nonvalid kernel if I'm given a kernel in the test?
Yes, it will be a valid kernel for all practical purposes, because ...
- 640
3
votes
Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?
I just found out the answer and it's actually pretty simple: while there is a good linear policy for the mountain car task, the value function itself is non-linear.
The state space of this task is ...
- 470
3
votes
Reinforcement learning without self-play
If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to ...
- 31
3
votes
Is TD-learning considered a model-based algorithm?
The previous answer is wrong. TD-learning is a model-free algorithm.
You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration.
Source: https:/...
- 248
3
votes
Accepted
How to setup the Bellman Equation as a linear system of equation
I found the full answer in a video by David Silver. The idea is easy enough.
The underlying matrix equation is
$$
v = R^\pi + \gamma P^\pi_{s, s'} v
$$
Where $v$ is the value function, $R^\pi$ is the ...
- 171
2
votes
Dyna-Q in non-deterministic domains
One approach is outlined in Tucker Balch's Machine Learning for Trading course on Udacity:
Learning T
Learning R
In a nutshell: create a $T_c$ table ("T count") that counts the number of times each ...
- 121
2
votes
Accepted
Can I have actions that, if taken, cannot be taken again in the following N timesteps?
This is not a limitation. Which actions are valid can depend on the current state of the system.
To enforce your requirement (changing speed at time T means you can't change again until time T+N), ...
D.W.♦
- 166k
2
votes
Accepted
Does an RL agent need to experience a complete episode in order to learn?
If the agent gets no feedback whatsoever and has no information about the structure of the search space (just blind search, hoping to get lucky), then no, it can't learn. It has nothing to learn from....
D.W.♦
- 166k
2
votes
Accepted
Approximate value iteration for continuous state space MDPs
There are two primary methods to deal with continuous state MDPs.
1. State-space discretization.
2. Value function approximation.
As for value function approximation, you can either go for a ...
- 66
2
votes
Exploration in reinforcement learning when state space is huge
In the Atari paper, an $\epsilon$-greedy strategy is used for state-space exploration. This means that the algorithm makes the deep network learn a greedy strategy to pick an action that maximises the ...
- 66
2
votes
What are the basics of CS i should know,before I start my journey into machine learning
Computer science is a very broad subject area, and many of its sub-disciplines have little or no overlap with others. For example, knowing the basics of operating systems design, compiler design or ...
- 1,589
2
votes
Accepted
Reinforcement learining basics - is it possible to deal with environment with random events
Yes, in general reinforcement learning algorithms can handle random events in the environment. Think about the q-learning update rule
$$ Q(s_t,a_t)= Q_{s_t, a_t} + \alpha \left(r_{t+1} + \gamma max_aQ(...
- 394
2
votes
Accepted
In reinforcement learning, does policy affect the maximization of the value?
Indeed in RL the policy $π$ does affect the maximization of the value function. This is a fundamental aspect of RL since the value function is defined w.r.t. a specific policy and different policies ...
- 330
1
vote
What is the relation between the compatibile features and the state features in Actor Critic Algorithm?
$\phi(s)$ is the representation of the state, and $\psi$ is the gradient (of the log) of the policy. Essentially, those two don't really have any connection, except for that the input to the policy is ...
- 11.7k
1
vote
Transition Function in MDP
I mean when some says that an agent, in real world grid, is going to step up by %80 and left/right by %10 each. How did he know that?
There are a couple of scenarios that could lead to an agent ...
- 394
1
vote
Accepted
What solution to apply for finding the optimal parameters?
It appears that you have a function $f:\mathbb{R}^4 \to \mathbb{R}$ and you want to find $x$ that maximizes $f(x)$, but you cannot compute $f$ directly; you can only obtain a noisy estimate of its ...
D.W.♦
- 166k
1
vote
Time efficient way to implement Multi-Armed-Bandits?
Store the values in a priority queue. Typically, in each iteration you will update the value for only a single arm, so you need to change the key of a single value in the priority queue, which can be ...
D.W.♦
- 166k
1
vote
Accepted
Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm
I don't think you wrote anything incorrect, it's just that you didn't finish your proof yet. How are you going to go from there to an actual bound on the regret? You're going to have to use the regret ...
- 4,189
1
vote
Accepted
Understanding On-policy First Visit Monte Carlo Control algorithm
The highlighted line basically says that time $t$ must be the first occurrence of the pair $(S_t, A_t)$ in the complete trajectory from $0$ up to and including $t$. This is why there is "first-visit" ...
- 295
1
vote
(DROP) Data Reduction Algorithm - How it works?
Algorithm 1 takes non-annotated data as input, so it is agnostic to the labels (shapes). In line 1, algorithm 1 computes clusters on the non-annotated data. The cluster assignments do not necessarily ...
- 163
1
vote
Is the credit assignment problem a well-posed one?
If you can do it well, it is useful. If you can accurately identify which move was to blame for bad outcomes -- or which one is to credit for good outcomes -- that is useful in helping you learn to ...
D.W.♦
- 166k
1
vote
Is deep learning appropriate to approximate dynamic programming problems?
A good start would be to understand what reinforcement learning is; and identify which problems are reinforcement learning problems and which aren't. Deep reinforcement learning is only relevant if ...
D.W.♦
- 166k
1
vote
Accepted
Existence of Optimal Policy for infinite-state MDPs
If the state set is Polish (e.g the set of all real numbers), finiteness of action spaces suffices to conclude that the MDP always has an optimal stationary policy - refer to "Markov Decision ...
- 36
1
vote
Eligibility trace and the role of gamma and lambda
At least for gamma I can answer it generally without looking in the book:
Gamma is commonly used as the discount factor. That is a number in (0, 1) which defines how much of the reward you got in ...
- 2,360
1
vote
Inverse Reinforcement Learning and Apprenticeship Learning differences
Apprenticeship Learning uses Inverse Reinforcement Learning and there is a subtle difference between the two. Inverse Reinforcement Learning tries to recover the reward function ($R^*$) from ...
- 66
1
vote
How can I improve this RL algorithm?
There's no reason to think that reinforcement learning would be effective here. Indeed, the surprise would be if it did work! Machine learning isn't a magic silver bullet. I wouldn't expect it to ...
D.W.♦
- 166k
1
vote
Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution
The best policy, known as bold play and figured out by Dubbins and Savage, is to always gamble as much as possible (but not more than is necessary to reach the goal). See for example an exposition by ...
- 279k
1
vote
Reinforcement learning without self-play
You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a ...
Only top scored, non community-wiki answers of a minimum length are eligible