Questions tagged [reinforcement-learning]
The reinforcement-learning tag has no usage guidance.
62
questions
1
vote
0answers
17 views
Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?
There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
0
votes
0answers
12 views
Logical inference as Markov decision process?
Are there efforts to consider forward or backward (theorem proving) logical inference process as Markov decision process in which the the action space is: 1) selection of the inference rule (e.g. from ...
1
vote
0answers
5 views
Is there any intersection in the applications of multi-agent reinforcement learning and more traditional branches of machine learning?
From my limited understanding, it seems like the structure of problems that multi agent reinforcement learning attempts to attack is quite different from the structure of problems in more traditional ...
0
votes
0answers
9 views
How to make use of opponent policy on designing ours in multi-agent learning
I'm currently working on a multi-agent reinforcement learning. The setting is a cooperative multi-agent system with a stationary assumption on the opponent policy. Suppose I have the model of the ...
0
votes
0answers
16 views
Why is bias not defined for non-stationary MDP policies?
The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:
$h(x, \phi) = \sum_{n = 0}^{\infty} E[r(x_n, \phi(x_n)) - w(x_n , \phi)]$
where $x$ is a ...
0
votes
0answers
10 views
Higher order reinforcement learning - that discovers new states and new actions (possibly in lifelong learning setting)?
Is there higher order reinforcement learning, that can not only find rewards (and hence optimal policy), bet that can also find the necessity to introduce new states and actions to better model the ...
1
vote
0answers
105 views
Is MCTS an appropriate method for this problem size (large action/state space)?
I'm doing a research on a finite horizon decision problem with $t=1,\dots,40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t)...
1
vote
0answers
20 views
Why is the objective in a multi-armed bandit problems to minimize (cumulative) regret?
In a multi-armed bandit, the objective is to maximize the expected cumulative reward. This objective is usually (equivalently?) stated in terms of expected cumulative regret.
Question: Why not just ...
1
vote
0answers
15 views
How to compute the sample update error?
In the book Reinforcement Learning An Introduction，Chapter 8.5，there is an example that compares the efficiency of expected and sample updates:
According to the author, "In this case, sample updates ...
1
vote
1answer
114 views
Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm
I was reading Elad Hazan's book on Online Convex Optimization(http://ocobook.cs.princeton.edu/OCObook.pdf) and am facing difficulty understanding the proof given for the No regret algorithm for MAB (...
1
vote
0answers
36 views
What does actually happens on tile hashing
I am going through Richard Sutton book about Reinforcement Learning and I just encountered the tile coding method. I understood pretty well the principles, however, at the very end of the section, ...
0
votes
1answer
67 views
Understanding On-policy First Visit Monte Carlo Control algorithm
I am going through the Monte Carlo methods, and it's going fine until now. However, I am actually studying the On-Policy First Visit Monte Carlo control for epsilon soft policies, which allows us to ...
3
votes
1answer
50 views
(DROP) Data Reduction Algorithm - How it works?
I am studing a PHD framework which the propose is to reduce the dataset with the most representative samples for training a classifier. Maybe I am loosing something, but I could not undestand a ...
0
votes
2answers
67 views
Is the credit assignment problem a well-posed one?
Credit assignment is the process of assigning credit (or blame) to a particular move in a sequence of moves (temporal credit assignment) or to a particular node (structural credit assignment) among ...
1
vote
1answer
169 views
Is deep learning appropriate to approximate dynamic programming problems?
I have a problem which can be completely solved using dynamic programming, but in a very intractable way (On^4, where n is around 1000). I won't get into the details of the problem since it's a bit ...
0
votes
0answers
56 views
Hessian in reinforcement learning
The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1].
The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues ...
0
votes
1answer
97 views
Existence of Optimal Policy for infinite-state MDPs
It is well-know from Puterman's book (1994) that in any finite-state MDP, if there exists an optimal policy, then that policy is stationary and deterministic. How about MDPs with continuous state ...
1
vote
0answers
50 views
Multi Arm Bandit (MAB) — Increasing reward function
In the general stochastic MAB model, the reward obtained at each trial is generally assumed to be independent of previous trials and obtained from some fixed (but unknown) distribution.
However, if ...
0
votes
0answers
35 views
Q-Learning Error Bounds
I have searched a lot for this, but apparently there is no result on calculating any bound on the error $||Q-Q^*||$ when I stop Q-learning after say $N$ iterations ($Q$ is the vector of Q-values at ...
1
vote
0answers
26 views
Markov Decision Process Optimal Policy
Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.
Say we have already computed values for the optimal $Q$-...
0
votes
2answers
37 views
Eligibility trace and the role of gamma and lambda
From R.Sutton's book, the eligibility trace update rule is:
$$
E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s)
$$
I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to ...
1
vote
1answer
42 views
Inverse Reinforcement Learning and Apprenticeship Learning differences
IRL as I understand is where the AI is given a policy from an "expert" and needs to solve for the reward function which best fits the policy.
And Apprenticeship Learning via IRL was explained as an ...
2
votes
1answer
563 views
Approximate value iteration for continuous state space MDPs
I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can ...
1
vote
1answer
116 views
Exploration in reinforcement learning when state space is huge
When the state-action space is huge I know I can use function approximators to generalize but how can I explore? Doing an exhaustive search seems very naive. What DeepMind for example did to explore ...
5
votes
1answer
176 views
Why does the effectiveness of my reinforcement based neural network recede after a while?
I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.
Typically, ...
1
vote
1answer
62 views
Bayesian updating for multivariate Gaussian
I am reading http://www.yisongyue.com/courses/cs159/lectures/LinUCB.pdf and come across this slide
What has been confusing me boils down to showing that multivariate Gaussian is conjugate to itself ...
6
votes
1answer
375 views
Machine learning approach to auction game
I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following:
I have a list of $N$ items which are ...
0
votes
1answer
44 views
How can I improve this RL algorithm?
This is the task: I have a unitary target matrix $T$, to decompose using matrices from a fixed (finite) universal set $\{M_i\}$, e.g. $T = M_6M_3M_9M_0$. The set is universal in the sense that one can ...
0
votes
1answer
34 views
Does an RL agent need to experience a complete episode in order to learn?
Suppose we have a game where there is no good measure of how "far" the target is (that we could use to nudge the agent towards the goal via the immediate returns).
If the agent gets only -1 reward ...
2
votes
1answer
68 views
Some questions on kernels and Reinforcement Learning
I've a test in a few days and I've a few issues with some of the subjects.
Let's start with kernels, basically I understood that a kernel needs to be positive semi-definite and symmetric in order to ...
2
votes
2answers
901 views
Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution
Currently reading a recent draft of Reinforcement Learning: An Introduction by Sutton and Barto. Really good book!
I was a bit confused by exercise 4.7 in chapter 4, section 4, page 93, (see ...
0
votes
0answers
189 views
How are punishment and negative reinforcement handled in reinforcement learning using neural net function approximation?
Punishment is reducing a behaviour due to bad outcome e.g. A cow stops touching an electric fence because it gets a shock.
Negative reinforcement is increasing a behaviour that reduces a bad outcome. ...
2
votes
1answer
201 views
Why do we use the log in gradient-based reinforcement algorithms?
I've been reading some papers on reinforcement learning.
$$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$
I often see expressions, similar to the above one, where the weights (denoted by $w$) are ...
3
votes
1answer
37 views
Can I have actions that, if taken, cannot be taken again in the following N timesteps?
I'm trying to train a load-balancing system with reinforcement-learning (RL) s.t. the incoming jobs are queued evenly at the available servers.
The system will not be able to directly dispatch the ...
2
votes
0answers
94 views
Q-learned policy differs from double-Q-learned policy
I implemented Q-learning and double-Q-learning as presented in Sutton's "Reinforcement Learning: An Introduction".
I test the algorithms on the OpenAI cliff walking gym and analyze the resulting ...
1
vote
0answers
63 views
Policy Gradient Methods
would be really thankful if someone could clear this up.
In reinforcement learning, When we use basis like the fourier or say polynomial basis, and say we have 4 actions, theta will then be a N x4 ...
2
votes
2answers
137 views
Reinforcement learning without self-play
Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?
For example, let'...
2
votes
1answer
2k views
Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?
I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except ...
2
votes
1answer
139 views
Reference request: Introduction to reinforcement learning with hand calculation examples
For me, the most difficulty when it comes to learning about reinforcement learning is that there is not much to learn in the sense that without running some algorithm, it is very difficult to get a ...
4
votes
1answer
43 views
When do the gradients exist in a Reinforcement Learning setting?
I am getting properly stuck into reinforcement learning and I am currently reading the review paper by Kober et al. (2013).
And there is one constant feature that I cannot get my head around, but ...
2
votes
0answers
57 views
How to best model multidimensional, continuous, non-convex “shape” as neural network?
I have:
set of n-dimensional points that I know are inside of the shape
n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity).
shape is pretty large in this n-...
2
votes
1answer
134 views
Reinforcement Learning - Q Learning
I am having trouble understanding the following problem and Q learning in general.
What I know so far about Q learning is that Q-learning is a model free method, i.e., it doesn’t need to learn P(s’|...
0
votes
3answers
199 views
Is TD-learning considered a model-based algorithm?
Differently from Sarsa and Q-learning, pure temporal difference learning (TD-learning) works with state value functions $V(s)$ and not state-action Q value functions $Q(s,a)$. It means that, in order ...
2
votes
0answers
60 views
Inconsistent results with Q-learning
I'm running a q-learning agent in tic tac toe against a minimax agent and measuring the tie rate (since you can never win against minimax). When I run 10,000 training games, I've found that there seem ...
1
vote
1answer
802 views
Difference between SNN RL and DNN RL?
In Reinfrocement Learning (RL) in Neural Networks (NNs), I've seen two approaches to Q-learning.
The first is to tile the state space with basis functions using Spiking Neural Networks (SNN) to ...
4
votes
1answer
236 views
reinforcement learning in gridworld with subgoals
Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.
There is a specific example ...
1
vote
1answer
215 views
Using a combination of spatial and non-spatial inputs for convolutional neural networks
I'm working on training a game AI using deep reinforcement learning to achieve specific examples based on pixel input and some additional state information.
Naturally, I'm using a convolutional ...
3
votes
1answer
680 views
How is the environment designed for testing a reinforcement learning algorithm?
I'm working on a project, and have a candidate algorithm which I'd like to test. Before I go any further, I need to get the hang of how to code the "structure" of the environment in which my system is ...
5
votes
1answer
673 views
Dyna-Q in non-deterministic domains
I've implemented the Dyna-Q reinforcement learning algorithm and it works perfectly on a discrete deterministic environment, the cliff. However, when applying it to a continuous environment (mountain ...
5
votes
1answer
2k views
Apprenticeship vs. imitation learning - what is the difference?
There are many sub-domains of reinforcement learning, two of which are apprenticeship and imitation learning - but that appears just to be two different names for the same thing?
