Questions tagged [reinforcement-learning]
The reinforcement-learning tag has no usage guidance.
1
vote
0answers
36 views
Is MCTS an appropriate method for this problem size (large action/state space)?
I'm doing a research on a finite horizon decision problem with $t=1,\dots,40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t)...
1
vote
0answers
18 views
Why is the objective in a multi-armed bandit problems to minimize (cumulative) regret?
In a multi-armed bandit, the objective is to maximize the expected cumulative reward. This objective is usually (equivalently?) stated in terms of expected cumulative regret.
Question: Why not just ...
1
vote
0answers
14 views
How to compute the sample update error?
In the book Reinforcement Learning An Introduction，Chapter 8.5，there is an example that compares the efficiency of expected and sample updates:
According to the author, "In this case, sample updates ...
0
votes
0answers
8 views
A question on Relative Entropy Inverse Reinforcement Learning
I am reading Boularias' Relative Entropy Inverse Reinforcment learning (http://proceedings.mlr.press/v15/boularias11a/boularias11a.pdf) and I do not understand a paragraph of the premise of the method ...
1
vote
1answer
85 views
Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm
I was reading Elad Hazan's book on Online Convex Optimization(http://ocobook.cs.princeton.edu/OCObook.pdf) and am facing difficulty understanding the proof given for the No regret algorithm for MAB (...
1
vote
0answers
25 views
What does actually happens on tile hashing
I am going through Richard Sutton book about Reinforcement Learning and I just encountered the tile coding method. I understood pretty well the principles, however, at the very end of the section, ...
0
votes
1answer
32 views
Understanding On-policy First Visit Monte Carlo Control algorithm
I am going through the Monte Carlo methods, and it's going fine until now. However, I am actually studying the On-Policy First Visit Monte Carlo control for epsilon soft policies, which allows us to ...
3
votes
1answer
46 views
(DROP) Data Reduction Algorithm - How it works?
I am studing a PHD framework which the propose is to reduce the dataset with the most representative samples for training a classifier. Maybe I am loosing something, but I could not undestand a ...
0
votes
2answers
36 views
Is the credit assignment problem a well-posed one?
Credit assignment is the process of assigning credit (or blame) to a particular move in a sequence of moves (temporal credit assignment) or to a particular node (structural credit assignment) among ...
1
vote
1answer
43 views
Is deep learning appropriate to approximate dynamic programming problems?
I have a problem which can be completely solved using dynamic programming, but in a very intractable way (On^4, where n is around 1000). I won't get into the details of the problem since it's a bit ...
0
votes
0answers
14 views
Is reinforcement learning the right tool for inferring graph structure from sequence?
Sorry that this is a complicated question - I will try to get to the essence of it.
I am interested in learning the structure of sequences of numbers in terms of a graph encoding specific types of ...
0
votes
0answers
37 views
Hessian in reinforcement learning
The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1].
The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues ...
0
votes
0answers
16 views
Reward surface in reinforcement learning
There is a remarkable paper [1] which explores geometry of neural network.
I believe this information is quite helpful in plethora of optimization methods. In reinforcement learning, the optimization ...
0
votes
1answer
64 views
Existence of Optimal Policy for infinite-state MDPs
It is well-know from Puterman's book (1994) that in any finite-state MDP, if there exists an optimal policy, then that policy is stationary and deterministic. How about MDPs with continuous state ...
1
vote
0answers
31 views
Multi Arm Bandit (MAB) — Increasing reward function
In the general stochastic MAB model, the reward obtained at each trial is generally assumed to be independent of previous trials and obtained from some fixed (but unknown) distribution.
However, if ...
0
votes
0answers
16 views
Relation between Q-learning and value iteration
Assume that Q-learning algorithm has already converged. Then can I say
$$
Q(s,a)=\mathbb{E}^{(s,a)}[R(s,a)] +\mathbb{E}^{\mathit{Pol}^*} [\sum\limits_{n=1}^{\infty} \gamma^n~ R(S_n,Pol^*(S_n))|S_1=s']
...
0
votes
0answers
37 views
stabilizing Q-Learning
I have this issue with Q-Learning that whenever I run it, it returns a different Q value for a certain state-action pair. Although, I am using decaying learning rate (e.g. 1/(time+1)) and gamma=0.99. ...
0
votes
0answers
29 views
Q-Learning Error Bounds
I have searched a lot for this, but apparently there is no result on calculating any bound on the error $||Q-Q^*||$ when I stop Q-learning after say $N$ iterations ($Q$ is the vector of Q-values at ...
0
votes
0answers
30 views
How Does Aproximate Reinforcement Learning Reduce State Space?
I was following reinforcement learning lecture from "CS188 Artificial Intelligence, Fall 2013". Here is the slide:
In the video, the lecturer says with approximate reinforcement learning we store ...
1
vote
0answers
21 views
Markov Decision Process Optimal Policy
Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.
Say we have already computed values for the optimal $Q$-...
0
votes
0answers
16 views
What is the correct loss fuction for this RL game?
I'm writing an RL algorithm similar to AlphaGo to play a single player game similar to solving a Rubik's cube and I'm trying to come up with a good loss function.
Starting from a random initial ...
0
votes
0answers
16 views
Simple Memory Mechanism in a highly customized recurrent neural network
I am currently implementing the experiment described in that paper:
"Emergence of grounded compositional language in Multi-Agent population" by Igor Mordatch and Pieter Abbeel, available here: https://...
0
votes
2answers
29 views
Eligibility trace and the role of gamma and lambda
From R.Sutton's book, the eligibility trace update rule is:
$$
E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s)
$$
I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to ...
0
votes
0answers
33 views
Meaning of saturation in reinforcement learning
In reinforcement learning, saturation happens at some point in time. I assume that it can mean either the problem is solved or the system is at a stage that it cannot do any better in solving the ...
1
vote
1answer
37 views
Inverse Reinforcement Learning and Apprenticeship Learning differences
IRL as I understand is where the AI is given a policy from an "expert" and needs to solve for the reward function which best fits the policy.
And Apprenticeship Learning via IRL was explained as an ...
3
votes
1answer
301 views
Approximate value iteration for continuous state space MDPs
I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can ...
1
vote
1answer
85 views
Exploration in reinforcement learning when state space is huge
When the state-action space is huge I know I can use function approximators to generalize but how can I explore? Doing an exhaustive search seems very naive. What DeepMind for example did to explore ...
5
votes
1answer
160 views
Why does the effectiveness of my reinforcement based neural network recede after a while?
I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.
Typically, ...
1
vote
1answer
59 views
Bayesian updating for multivariate Gaussian
I am reading http://www.yisongyue.com/courses/cs159/lectures/LinUCB.pdf and come across this slide
What has been confusing me boils down to showing that multivariate Gaussian is conjugate to itself ...
5
votes
1answer
296 views
Machine learning approach to auction game
I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following:
I have a list of $N$ items which are ...
0
votes
1answer
39 views
How can I improve this RL algorithm?
This is the task: I have a unitary target matrix $T$, to decompose using matrices from a fixed (finite) universal set $\{M_i\}$, e.g. $T = M_6M_3M_9M_0$. The set is universal in the sense that one can ...
0
votes
1answer
34 views
Does an RL agent need to experience a complete episode in order to learn?
Suppose we have a game where there is no good measure of how "far" the target is (that we could use to nudge the agent towards the goal via the immediate returns).
If the agent gets only -1 reward ...
2
votes
1answer
65 views
Some questions on kernels and Reinforcement Learning
I've a test in a few days and I've a few issues with some of the subjects.
Let's start with kernels, basically I understood that a kernel needs to be positive semi-definite and symmetric in order to ...
2
votes
2answers
669 views
Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution
Currently reading a recent draft of Reinforcement Learning: An Introduction by Sutton and Barto. Really good book!
I was a bit confused by exercise 4.7 in chapter 4, section 4, page 93, (see ...
0
votes
0answers
150 views
How are punishment and negative reinforcement handled in reinforcement learning using neural net function approximation?
Punishment is reducing a behaviour due to bad outcome e.g. A cow stops touching an electric fence because it gets a shock.
Negative reinforcement is increasing a behaviour that reduces a bad outcome. ...
0
votes
1answer
67 views
why take log in gradient-based reinforcement algorithms
$$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$
I've been studying recently reading some papers on reinforcement learning. I often see expressions similar to above where the weights are updated ...
3
votes
1answer
36 views
Can I have actions that, if taken, cannot be taken again in the following N timesteps?
I'm trying to train a load-balancing system with reinforcement-learning (RL) s.t. the incoming jobs are queued evenly at the available servers.
The system will not be able to directly dispatch the ...
3
votes
0answers
83 views
Q-learned policy differs from double-Q-learned policy
I implemented Q-learning and double-Q-learning as presented in Sutton's "Reinforcement Learning: An Introduction".
I test the algorithms on the OpenAI cliff walking gym and analyze the resulting ...
1
vote
0answers
61 views
Policy Gradient Methods
would be really thankful if someone could clear this up.
In reinforcement learning, When we use basis like the fourier or say polynomial basis, and say we have 4 actions, theta will then be a N x4 ...
2
votes
2answers
131 views
Reinforcement learning without self-play
Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?
For example, let'...
2
votes
1answer
2k views
Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?
I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except ...
2
votes
1answer
129 views
Reference request: Introduction to reinforcement learning with hand calculation examples
For me, the most difficulty when it comes to learning about reinforcement learning is that there is not much to learn in the sense that without running some algorithm, it is very difficult to get a ...
4
votes
1answer
40 views
When do the gradients exist in a Reinforcement Learning setting?
I am getting properly stuck into reinforcement learning and I am currently reading the review paper by Kober et al. (2013).
And there is one constant feature that I cannot get my head around, but ...
3
votes
0answers
55 views
How to best model multidimensional, continuous, non-convex “shape” as neural network?
I have:
set of n-dimensional points that I know are inside of the shape
n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity).
shape is pretty large in this n-...
2
votes
1answer
121 views
Reinforcement Learning - Q Learning
I am having trouble understanding the following problem and Q learning in general.
What I know so far about Q learning is that Q-learning is a model free method, i.e., it doesn’t need to learn P(s’|...
0
votes
3answers
161 views
Is TD-learning considered a model-based algorithm?
Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order ...
3
votes
0answers
56 views
Inconsistent results with Q-learning
I'm running a q-learning agent in tic tac toe against a minimax agent and measuring the tie rate (since you can never win against minimax). When I run 10,000 training games, I've found that there seem ...
1
vote
1answer
573 views
Difference between SNN RL and DNN RL?
In Reinfrocement Learning (RL) in Neural Networks (NNs), I've seen two approaches to Q-learning.
The first is to tile the state space with basis functions using Spiking Neural Networks (SNN) to ...
4
votes
1answer
211 views
reinforcement learning in gridworld with subgoals
Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.
There is a specific example ...
1
vote
1answer
152 views
Using a combination of spatial and non-spatial inputs for convolutional neural networks
I'm working on training a game AI using deep reinforcement learning to achieve specific examples based on pixel input and some additional state information.
Naturally, I'm using a convolutional ...
