# Tag Info

Accepted

We often take the logarithm because: Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of ...
• 163k

### Some questions on kernels and Reinforcement Learning

Is that valid? My question is how can I distinguish between a valid kernel and a nonvalid kernel if I'm given a kernel in the test? Yes, it will be a valid kernel for all practical purposes, because ...
• 640
Accepted

### Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The question, as written, certainly suggests that it's a "good policy" to bet a=50 for s=50 , while not "betting it all" with a=49 for s=51. However, in an e-mail discussion surrounding this (here), ...
• 146

### Reinforcement learning without self-play

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to ...
• 31

### Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?

I just found out the answer and it's actually pretty simple: while there is a good linear policy for the mountain car task, the value function itself is non-linear. The state space of this task is ...
• 471

### Is TD-learning considered a model-based algorithm?

The previous answer is wrong. TD-learning is a model-free algorithm. You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration. Source: https:/...
• 248
Accepted

### How to setup the Bellman Equation as a linear system of equation

I found the full answer in a video by David Silver. The idea is easy enough. The underlying matrix equation is $$v = R^\pi + \gamma P^\pi_{s, s'} v$$ Where $v$ is the value function, $R^\pi$ is the ...
• 171

### Is TD-learning considered a model-based algorithm?

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is ...
• 21

### Dyna-Q in non-deterministic domains

One approach is outlined in Tucker Balch's Machine Learning for Trading course on Udacity: Learning T Learning R In a nutshell: create a $T_c$ table ("T count") that counts the number of times each ...
• 121
Accepted

### Reference request: Introduction to reinforcement learning with hand calculation examples

Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents, fully available online, has one such example for Q-learning. Sutton & Barto's Reinforcement Learning: An ...
• 171
Accepted

### When do the gradients exist in a Reinforcement Learning setting?

The gradient doesn't exist / isn't well-defined for non-differentiable functions. What they mean by that statement is that there is an analogous version of gradients that can be used, instead of the ...
• 163k
Accepted

### Can I have actions that, if taken, cannot be taken again in the following N timesteps?

This is not a limitation. Which actions are valid can depend on the current state of the system. To enforce your requirement (changing speed at time T means you can't change again until time T+N), ...
• 163k
Accepted

### Does an RL agent need to experience a complete episode in order to learn?

If the agent gets no feedback whatsoever and has no information about the structure of the search space (just blind search, hoping to get lucky), then no, it can't learn. It has nothing to learn from....
• 163k

### What are the basics of CS i should know,before I start my journey into machine learning

Computer science is a very broad subject area, and many of its sub-disciplines have little or no overlap with others. For example, knowing the basics of operating systems design, compiler design or ...
• 1,589
Accepted

### Approximate value iteration for continuous state space MDPs

There are two primary methods to deal with continuous state MDPs. 1. State-space discretization. 2. Value function approximation. As for value function approximation, you can either go for a ...

### Exploration in reinforcement learning when state space is huge

In the Atari paper, an $\epsilon$-greedy strategy is used for state-space exploration. This means that the algorithm makes the deep network learn a greedy strategy to pick an action that maximises the ...
Accepted

### Reinforcement learining basics - is it possible to deal with environment with random events

Yes, in general reinforcement learning algorithms can handle random events in the environment. Think about the q-learning update rule  Q(s_t,a_t)= Q_{s_t, a_t} + \alpha \left(r_{t+1} + \gamma max_aQ(...
• 394
1 vote
Accepted

### What solution to apply for finding the optimal parameters?

It appears that you have a function $f:\mathbb{R}^4 \to \mathbb{R}$ and you want to find $x$ that maximizes $f(x)$, but you cannot compute $f$ directly; you can only obtain a noisy estimate of its ...
• 163k
1 vote

### Time efficient way to implement Multi-Armed-Bandits?

Store the values in a priority queue. Typically, in each iteration you will update the value for only a single arm, so you need to change the key of a single value in the priority queue, which can be ...
• 163k
1 vote

### How can I improve this RL algorithm?

There's no reason to think that reinforcement learning would be effective here. Indeed, the surprise would be if it did work! Machine learning isn't a magic silver bullet. I wouldn't expect it to ...
• 163k
1 vote
Accepted

### Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm

I don't think you wrote anything incorrect, it's just that you didn't finish your proof yet. How are you going to go from there to an actual bound on the regret? You're going to have to use the regret ...
• 4,149
1 vote
Accepted

### Understanding On-policy First Visit Monte Carlo Control algorithm

The highlighted line basically says that time $t$ must be the first occurrence of the pair $(S_t, A_t)$ in the complete trajectory from $0$ up to and including $t$. This is why there is "first-visit" ...
1 vote

### (DROP) Data Reduction Algorithm - How it works?

Algorithm 1 takes non-annotated data as input, so it is agnostic to the labels (shapes). In line 1, algorithm 1 computes clusters on the non-annotated data. The cluster assignments do not necessarily ...
• 163
1 vote

### Is the credit assignment problem a well-posed one?

If you can do it well, it is useful. If you can accurately identify which move was to blame for bad outcomes -- or which one is to credit for good outcomes -- that is useful in helping you learn to ...
• 163k
1 vote

### Is deep learning appropriate to approximate dynamic programming problems?

A good start would be to understand what reinforcement learning is; and identify which problems are reinforcement learning problems and which aren't. Deep reinforcement learning is only relevant if ...
• 163k
1 vote
Accepted

### Existence of Optimal Policy for infinite-state MDPs

If the state set is Polish (e.g the set of all real numbers), finiteness of action spaces suffices to conclude that the MDP always has an optimal stationary policy - refer to "Markov Decision ...
• 36
1 vote

### Eligibility trace and the role of gamma and lambda

At least for gamma I can answer it generally without looking in the book: Gamma is commonly used as the discount factor. That is a number in (0, 1) which defines how much of the reward you got in ...
• 2,360
1 vote

### Inverse Reinforcement Learning and Apprenticeship Learning differences

Apprenticeship Learning uses Inverse Reinforcement Learning and there is a subtle difference between the two. Inverse Reinforcement Learning tries to recover the reward function ($R^*$) from ...
1 vote

### Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

The best policy, known as bold play and figured out by Dubbins and Savage, is to always gamble as much as possible (but not more than is necessary to reach the goal). See for example an exposition by ...
• 278k
1 vote

### Reinforcement learning without self-play

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a ...

Only top scored, non community-wiki answers of a minimum length are eligible