Questions tagged [reinforcement-learning]
The reinforcement-learning tag has no usage guidance.
0
votes
0answers
14 views
Why is bias not defined for non-stationary MDP policies?
The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:
$h(x, \phi) = \sum_{n = 0}^{\infty} E[r(x_n, \phi(x_n)) - w(x_n , \phi)]$
where $x$ is a ...
0
votes
0answers
8 views
Higher order reinforcement learning - that discovers new states and new actions (possibly in lifelong learning setting)?
Is there higher order reinforcement learning, that can not only find rewards (and hence optimal policy), bet that can also find the necessity to introduce new states and actions to better model the ...
1
vote
0answers
87 views
Is MCTS an appropriate method for this problem size (large action/state space)?
I'm doing a research on a finite horizon decision problem with $t=1,\dots,40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t)...
1
vote
0answers
18 views
Why is the objective in a multi-armed bandit problems to minimize (cumulative) regret?
In a multi-armed bandit, the objective is to maximize the expected cumulative reward. This objective is usually (equivalently?) stated in terms of expected cumulative regret.
Question: Why not just ...
1
vote
0answers
15 views
How to compute the sample update error?
In the book Reinforcement Learning An Introduction，Chapter 8.5，there is an example that compares the efficiency of expected and sample updates:
According to the author, "In this case, sample updates ...
1
vote
1answer
108 views
Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm
I was reading Elad Hazan's book on Online Convex Optimization(http://ocobook.cs.princeton.edu/OCObook.pdf) and am facing difficulty understanding the proof given for the No regret algorithm for MAB (...
1
vote
0answers
32 views
What does actually happens on tile hashing
I am going through Richard Sutton book about Reinforcement Learning and I just encountered the tile coding method. I understood pretty well the principles, however, at the very end of the section, ...
0
votes
1answer
58 views
Understanding On-policy First Visit Monte Carlo Control algorithm
I am going through the Monte Carlo methods, and it's going fine until now. However, I am actually studying the On-Policy First Visit Monte Carlo control for epsilon soft policies, which allows us to ...
3
votes
1answer
49 views
(DROP) Data Reduction Algorithm - How it works?
I am studing a PHD framework which the propose is to reduce the dataset with the most representative samples for training a classifier. Maybe I am loosing something, but I could not undestand a ...
0
votes
2answers
58 views
Is the credit assignment problem a well-posed one?
Credit assignment is the process of assigning credit (or blame) to a particular move in a sequence of moves (temporal credit assignment) or to a particular node (structural credit assignment) among ...
1
vote
1answer
117 views
Is deep learning appropriate to approximate dynamic programming problems?
I have a problem which can be completely solved using dynamic programming, but in a very intractable way (On^4, where n is around 1000). I won't get into the details of the problem since it's a bit ...
0
votes
0answers
19 views
Is reinforcement learning the right tool for inferring graph structure from sequence?
Sorry that this is a complicated question - I will try to get to the essence of it.
I am interested in learning the structure of sequences of numbers in terms of a graph encoding specific types of ...
0
votes
0answers
50 views
Hessian in reinforcement learning
The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1].
The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues ...
0
votes
0answers
18 views
Reward surface in reinforcement learning
There is a remarkable paper [1] which explores geometry of neural network.
I believe this information is quite helpful in plethora of optimization methods. In reinforcement learning, the optimization ...
0
votes
1answer
89 views
Existence of Optimal Policy for infinite-state MDPs
It is well-know from Puterman's book (1994) that in any finite-state MDP, if there exists an optimal policy, then that policy is stationary and deterministic. How about MDPs with continuous state ...
1
vote
0answers
44 views
Multi Arm Bandit (MAB) — Increasing reward function
In the general stochastic MAB model, the reward obtained at each trial is generally assumed to be independent of previous trials and obtained from some fixed (but unknown) distribution.
However, if ...
0
votes
0answers
18 views
Relation between Q-learning and value iteration
Assume that Q-learning algorithm has already converged. Then can I say
$$
Q(s,a)=\mathbb{E}^{(s,a)}[R(s,a)] +\mathbb{E}^{\mathit{Pol}^*} [\sum\limits_{n=1}^{\infty} \gamma^n~ R(S_n,Pol^*(S_n))|S_1=s']
...
0
votes
0answers
42 views
stabilizing Q-Learning
I have this issue with Q-Learning that whenever I run it, it returns a different Q value for a certain state-action pair. Although, I am using decaying learning rate (e.g. 1/(time+1)) and gamma=0.99. ...
0
votes
0answers
32 views
Q-Learning Error Bounds
I have searched a lot for this, but apparently there is no result on calculating any bound on the error $||Q-Q^*||$ when I stop Q-learning after say $N$ iterations ($Q$ is the vector of Q-values at ...
0
votes
0answers
43 views
How Does Aproximate Reinforcement Learning Reduce State Space?
I was following reinforcement learning lecture from "CS188 Artificial Intelligence, Fall 2013". Here is the slide:
In the video, the lecturer says with approximate reinforcement learning we store ...
1
vote
0answers
25 views
Markov Decision Process Optimal Policy
Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.
Say we have already computed values for the optimal $Q$-...
0
votes
2answers
32 views
Eligibility trace and the role of gamma and lambda
From R.Sutton's book, the eligibility trace update rule is:
$$
E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s)
$$
I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to ...
1
vote
1answer
41 views
Inverse Reinforcement Learning and Apprenticeship Learning differences
IRL as I understand is where the AI is given a policy from an "expert" and needs to solve for the reward function which best fits the policy.
And Apprenticeship Learning via IRL was explained as an ...
3
votes
1answer
492 views
Approximate value iteration for continuous state space MDPs
I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can ...
1
vote
1answer
106 views
Exploration in reinforcement learning when state space is huge
When the state-action space is huge I know I can use function approximators to generalize but how can I explore? Doing an exhaustive search seems very naive. What DeepMind for example did to explore ...
5
votes
1answer
172 views
Why does the effectiveness of my reinforcement based neural network recede after a while?
I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.
Typically, ...
1
vote
1answer
59 views
Bayesian updating for multivariate Gaussian
I am reading http://www.yisongyue.com/courses/cs159/lectures/LinUCB.pdf and come across this slide
What has been confusing me boils down to showing that multivariate Gaussian is conjugate to itself ...
6
votes
1answer
341 views
Machine learning approach to auction game
I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following:
I have a list of $N$ items which are ...
0
votes
1answer
42 views
How can I improve this RL algorithm?
This is the task: I have a unitary target matrix $T$, to decompose using matrices from a fixed (finite) universal set $\{M_i\}$, e.g. $T = M_6M_3M_9M_0$. The set is universal in the sense that one can ...
0
votes
1answer
34 views
Does an RL agent need to experience a complete episode in order to learn?
Suppose we have a game where there is no good measure of how "far" the target is (that we could use to nudge the agent towards the goal via the immediate returns).
If the agent gets only -1 reward ...
2
votes
1answer
68 views
Some questions on kernels and Reinforcement Learning
I've a test in a few days and I've a few issues with some of the subjects.
Let's start with kernels, basically I understood that a kernel needs to be positive semi-definite and symmetric in order to ...
2
votes
2answers
842 views
Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution
Currently reading a recent draft of Reinforcement Learning: An Introduction by Sutton and Barto. Really good book!
I was a bit confused by exercise 4.7 in chapter 4, section 4, page 93, (see ...
0
votes
0answers
176 views
How are punishment and negative reinforcement handled in reinforcement learning using neural net function approximation?
Punishment is reducing a behaviour due to bad outcome e.g. A cow stops touching an electric fence because it gets a shock.
Negative reinforcement is increasing a behaviour that reduces a bad outcome. ...
2
votes
1answer
142 views
Why do we use the log in gradient-based reinforcement algorithms?
I've been reading some papers on reinforcement learning.
$$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$
I often see expressions, similar to the above one, where the weights (denoted by $w$) are ...
3
votes
1answer
36 views
Can I have actions that, if taken, cannot be taken again in the following N timesteps?
I'm trying to train a load-balancing system with reinforcement-learning (RL) s.t. the incoming jobs are queued evenly at the available servers.
The system will not be able to directly dispatch the ...
3
votes
0answers
93 views
Q-learned policy differs from double-Q-learned policy
I implemented Q-learning and double-Q-learning as presented in Sutton's "Reinforcement Learning: An Introduction".
I test the algorithms on the OpenAI cliff walking gym and analyze the resulting ...
1
vote
0answers
62 views
Policy Gradient Methods
would be really thankful if someone could clear this up.
In reinforcement learning, When we use basis like the fourier or say polynomial basis, and say we have 4 actions, theta will then be a N x4 ...
2
votes
2answers
132 views
Reinforcement learning without self-play
Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?
For example, let'...
2
votes
1answer
2k views
Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?
I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except ...
2
votes
1answer
133 views
Reference request: Introduction to reinforcement learning with hand calculation examples
For me, the most difficulty when it comes to learning about reinforcement learning is that there is not much to learn in the sense that without running some algorithm, it is very difficult to get a ...
4
votes
1answer
40 views
When do the gradients exist in a Reinforcement Learning setting?
I am getting properly stuck into reinforcement learning and I am currently reading the review paper by Kober et al. (2013).
And there is one constant feature that I cannot get my head around, but ...
3
votes
0answers
57 views
How to best model multidimensional, continuous, non-convex “shape” as neural network?
I have:
set of n-dimensional points that I know are inside of the shape
n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity).
shape is pretty large in this n-...
2
votes
1answer
125 views
Reinforcement Learning - Q Learning
I am having trouble understanding the following problem and Q learning in general.
What I know so far about Q learning is that Q-learning is a model free method, i.e., it doesn’t need to learn P(s’|...
0
votes
3answers
187 views
Is TD-learning considered a model-based algorithm?
Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order ...
3
votes
0answers
59 views
Inconsistent results with Q-learning
I'm running a q-learning agent in tic tac toe against a minimax agent and measuring the tie rate (since you can never win against minimax). When I run 10,000 training games, I've found that there seem ...
1
vote
1answer
719 views
Difference between SNN RL and DNN RL?
In Reinfrocement Learning (RL) in Neural Networks (NNs), I've seen two approaches to Q-learning.
The first is to tile the state space with basis functions using Spiking Neural Networks (SNN) to ...
4
votes
1answer
230 views
reinforcement learning in gridworld with subgoals
Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.
There is a specific example ...
1
vote
1answer
188 views
Using a combination of spatial and non-spatial inputs for convolutional neural networks
I'm working on training a game AI using deep reinforcement learning to achieve specific examples based on pixel input and some additional state information.
Naturally, I'm using a convolutional ...
3
votes
1answer
626 views
How is the environment designed for testing a reinforcement learning algorithm?
I'm working on a project, and have a candidate algorithm which I'd like to test. Before I go any further, I need to get the hang of how to code the "structure" of the environment in which my system is ...
5
votes
1answer
657 views
Dyna-Q in non-deterministic domains
I've implemented the Dyna-Q reinforcement learning algorithm and it works perfectly on a discrete deterministic environment, the cliff. However, when applying it to a continuous environment (mountain ...
