# Tag Info

6

The discount factor does not represent the likelihood to reach the state $s′$from the state $s$. That would be $p(s'|s,a)$, which is not used in Q-Learning, since it is model-free (only model-based reinforcement learning methods use those transition probabilities). The discount factor $γ$ is a hyperparameter tuned by the user which represents how much ...

5

We often take the logarithm because: Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of maximizing the likelihood directly and the result will be equivalent. The logarithm converts multiplication to addition, and the derivative of a sum is "nicer" than the ...

5

In general you can't change the state and action spaces (rows and columns of your reward matrix) since that changes the underlying model that you are sampling with Q-learning. It'd be similar to changing the population you're sampling when statistically sampling. Hence, you have to define a set of actions and states that don't change in order to use Q-...

5

When you're driving of your car and you turn the wheel, is that a discrete or a continuous action? It's continuous, because you can control how much you turn the wheel. How much do you press the gas pedal? That's a continuous input. This leads to a continuous action space: e.g., for each positive real number $x$ in some range, "turn the wheel $x$ degrees ...

4

The difference between the two tasks really comes down to the level of continuity assumed in the models of the problem. In adaptive control, continuity is assumed at all levels; the problem space and the actions to be executed are all continuous. In hierarchical reinforcement learning, although the problem space is continuous, the actions to execute upon ...

4

The assumptions for Q-learning's convergence proof require that: [...] all state-action pairs be visited infinitely often So the convergence is understood in a mathematical sense - the Q-function converges to its fixed-point as the number of steps approaches infinity. In practice, even after your Q-function will be good enough to induce an optimal ...

4

In general, yes, they are the same thing, which means to learn from demonstration (LfD). But, usually, apprenticeship learning is mentioned in the context of "Apprenticeship learning via inverse reinforcement learning" (IRL). Both methods learn from demonstration, but they learn different things: Imitation learning (a.k.a. behavioral cloning) will ...

3

Is that valid? My question is how can I distinguish between a valid kernel and a nonvalid kernel if I'm given a kernel in the test? Yes, it will be a valid kernel for all practical purposes, because kernel functions measure the similarity of some points relative to other ones (that's why they are also called similarity functions). Making your kernel ...

3

If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to generate many more training examples on which to train a policy by selecting new action combinations. This may result in unexpected behaviour if you do not have ...

3

The question, as written, certainly suggests that it's a "good policy" to bet a=50 for s=50 , while not "betting it all" with a=49 for s=51. However, in an e-mail discussion surrounding this (here), Daniel Loeb asks a similar question and notes, "in particular, the simple policy of always making the largest allowed bet is optimal", which would suggest that ...

3

The previous answer is wrong. TD-learning is a model-free algorithm. You compute $V_{\pi}(s)$ for a fixed policy $\pi$, then you update your policy based on $V$ a la policy iteration. Source: https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node60.html

3

I just found out the answer and it's actually pretty simple: while there is a good linear policy for the mountain car task, the value function itself is non-linear. The state space of this task is like a spiral, and there is no linear approximation possible even for a mediocre value function. If you increase the value for going right and reaching the top, ...

2

Poole and Mackworth's Artificial Intelligence: Foundations of Computational Agents, fully available online, has one such example for Q-learning. Sutton & Barto's Reinforcement Learning: An Introduction, also online, has a few tiny iterative examples here and there. Isbell & Littman's udacity course on RL has examples, quizzes, and programming ...

2

The gradient doesn't exist / isn't well-defined for non-differentiable functions. What they mean by that statement is that there is an analogous version of gradients that can be used, instead of the gradient. Discrete functions In the discrete case, finite differences are the discrete version of derivatives. The derivative of a single-variable continuous ...

2

One approach is outlined in Tucker Balch's Machine Learning for Trading course on Udacity: Learning T Learning R In a nutshell: create a $T_c$ table ("T count") that counts the number of times each subsequent state is reached after taking an action in a given state. Initialize all values to 0.00001 to avoid division by 0 errors. While executing QLearning, ...

2

This is not a limitation. Which actions are valid can depend on the current state of the system. To enforce your requirement (changing speed at time T means you can't change again until time T+N), you can add additional information to the state: basically, add how long ago was the last change (where if it's more than N steps ago, you don't keep track of ...

2

If the agent gets no feedback whatsoever and has no information about the structure of the search space (just blind search, hoping to get lucky), then no, it can't learn. It has nothing to learn from. In your scenario, it has no information to enable it to improve: the feedback is always the same -- always an unending stream of -1,-1,-1,-1,-1,... unless it ...

2

In the Atari paper, an $\epsilon$-greedy strategy is used for state-space exploration. This means that the algorithm makes the deep network learn a greedy strategy to pick an action that maximises the $Q$-value of the current state ($a = max_a Q(s, a; \theta)$. At each time-step in a game, the agent picks an action as per this greedy strategy with ...

2

There are two primary methods to deal with continuous state MDPs. 1. State-space discretization. 2. Value function approximation. As for value function approximation, you can either go for a deterministic/stochastic black-box model or opt for fitted value iteration. You can find the algorithm for the same in pages 10-13 of this link.

2

Computer science is a very broad subject area, and many of its sub-disciplines have little or no overlap with others. For example, knowing the basics of operating systems design, compiler design or microprocessor design are unlikely to help you make progress in machine learning (although each one is an interesting topic in its own right). Machine learning ...

2

Yes, in general reinforcement learning algorithms can handle random events in the environment. Think about the q-learning update rule $$Q(s_t,a_t)= Q_{s_t, a_t} + \alpha \left(r_{t+1} + \gamma max_aQ(s_{t+1},a)\right)$$ But remember $$Q(s_{t+1},a) = \mathbb{E}_\pi\left( \sum_{i=t+2}^T\gamma^{i-1}r_{i} \right)$$ that is the expected value of the discounted ...

1

There's no reason to think that reinforcement learning would be effective here. Indeed, the surprise would be if it did work! Machine learning isn't a magic silver bullet. I wouldn't expect it to be of any assistance here; the problem relates to deep questions about the structure of the general linear group, which might be quite challenging for your ...

1

The best policy, known as bold play and figured out by Dubbins and Savage, is to always gamble as much as possible (but not more than is necessary to reach the goal). See for example an exposition by Kyle Siegrist, How to gamble if you must.

1

You might want to look at Least squares policy iteration. These algorithms can incorporate data sets of the form and is an offline method. These may help you incorporate your past experiences in a fruitful manner.

1

At t=1, you just took action a = d at state s = S and ended in s' = S again with 0 reward. So: $Q(S,d) = (1 - 1/2)(1) + 1/2[0 + (1)(1)] = 1$ ($maxQ(s',a')$ is the maximum value you can get from state S in this case, which is 1) So nothing changes. At t=2, it is exactly the same as t=1, so nothing changes again. At t=3, we just took action p at s = S and ...

1

For prediction problems, in TD(0) we don't need to (greedily) pick a action based on $V(s)$, so it's not a model-based algo. As for control problems, Q-learning & Sarsa are both TD algos, and it's obivious that they are both model-free control algos. However, if you want to use $V(s)$ in a control problem -- it's definitely model-based and that is ...

1

Yes. TD learning is a model-based RL algorithm because you can't extract a policy from $V(s)$ without having a transition model $T(s,a,s')$ to sample next states from. I.e., knowing $V^*(s)$ is useless in terms of decision making if you don't also have a model of the MDP. The reason Q-learning and SARSA are considered model-free is because you don't need ...

1

Daniel Rasmussen replied via email: SNNs take less long to train This really depends on the training method, and the implementation. If you just implemented an abstract DNN approach to solve the same task as is being solved in those NEF RL papers, it would be a pretty simple network and would train quite fast (almost certainly faster ...

1

Yes. One way is to start with several convolutional layers, and then end with one or two fully connected layers. You can add the other features as inputs to the first of those fully connected layers. This is basically serial composition of convolutional layers then fully connected layers. There are other options for network architectures, but that'd be a ...

1

In reinforcement learning, the "environment" is typically a set of states the "agent" is attempting to influence via its choice of "actions". For example, in "Reinforcement learning design for cancer clinical trials" by Zhao, Kosorok, and Zeng (2009), ..."states" may represent individual patient covariates and "actions" can be denoted by various ...

Only top voted, non community-wiki answers of a minimum length are eligible