Questions tagged [reinforcement-learning]
The reinforcement-learning tag has no usage guidance.
45 questions with no upvoted or accepted answers
6
votes
1
answer
641
views
Machine learning approach to auction game
I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following:
I have a list of $N$ items which are ...
- 161
5
votes
1
answer
420
views
reinforcement learning in gridworld with subgoals
Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.
There is a specific example ...
- 273
3
votes
1
answer
375
views
Policy dependent on initial state distribution in finite horizon MDPs
Consider an MDP defined as the tuple $\langle S,A,R,P,\mu,\lambda\rangle$ where $S$ is the state space, $A$ the action space, $R:S\times A\times S\to\mathbb{R}$ the reward function, $P$ the transition ...
- 371
3
votes
0
answers
561
views
Reinforcement learning - state space and action space
I am working on a reinforcement learning strategy for parameter control of a local search heuristic.
The complete state for this RL problem can be defined as $S = \{s, p\}$, where $s$ and $p$ ...
- 31
2
votes
0
answers
34
views
Where to find the current state of the art performance of Deep RL algorithms?
Recently, I had an idea of a novel Deep RL algorithm that might perform better than existing algorithms such as DQN, TRPO, PPO, etc.
However, I do not know of a website or a research paper that might ...
- 161
2
votes
0
answers
225
views
Discrete action space for MADDPG
I am trying to apply MADDPG, a policy gradient algorithm that uses centralized training and decentralized execution, to a project. In the work of Lowe et al., the actor returns a pmf for a discrete ...
2
votes
0
answers
294
views
Relationship between dynamic programming and reinforcement learning
I wasn't sure whether to post this here or in the ai stack exchange - please let me know if i need to move my post elsewhere)
I have been learning about how dynamic programming can be used as a tool ...
- 121
2
votes
0
answers
173
views
Q-learned policy differs from double-Q-learned policy
I implemented Q-learning and double-Q-learning as presented in Sutton's "Reinforcement Learning: An Introduction".
I test the algorithms on the OpenAI cliff walking gym and analyze the resulting ...
- 153
2
votes
0
answers
69
views
How to best model multidimensional, continuous, non-convex "shape" as neural network?
I have:
set of n-dimensional points that I know are inside of the shape
n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity).
shape is pretty large in this n-...
- 131
1
vote
1
answer
19
views
In multi-agent RL stochastic games (multi-agent version of an MDP), why does the Nash equilibrium policy only depend on the state, not history?
I am studying from the MARL textbook by Albrecht, Christianos and Schäfer. They define a stochastic game in Sec 3.3 as the multi-agent version of an MDP. In Fig 3.3 (pg 50) they give an intuition for ...
1
vote
0
answers
91
views
DeepMind Alphadev: How did it use Reinforcement Learning to reduce the search space?
Google DeepMind recently published a new paper which describes how they used a reinforcement learning to discover faster sorting algorithms. A summary of the paper is here and the paper is here.
It ...
- 11
1
vote
0
answers
144
views
Tabular Meta-Learning in RL
There are various meta-learning algorithms in RL that are proposed for settings when we have a (deep) neural network and the policy (or the value function) are parameterized as such.
Can these methods ...
- 175
1
vote
0
answers
60
views
Long and short memory in reinforcement learning Connect 4 AI
I'm writing an AI based on reinforcement learning to play Connect 4. That's my second bot and attempt to RNN and AI (first was copy a code of snake RNN AI from youtube) and I'm looking for some ...
- 11
1
vote
0
answers
14
views
Evaluating the safety of deep RL algorithms
Is there any free/open-source environment, tasks, or dataset for evaluating deep RL algorithms in terms of safety?
all available environments (like openAI's) are environments for simple games. These ...
- 161
1
vote
0
answers
39
views
Deep RL for healthcare: existing benchmarking datasets or environments
I am currently a Ph.D. student in the computer science department, I was given the subject of Deep RL for Healthcare. However, after lots of research on the internet, I could not find any free dataset ...
- 161
1
vote
0
answers
16
views
Reinforcement learning with 0 rewards and costs
Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
- 11
1
vote
0
answers
27
views
Resources on Dynamic Programming with Indefinite Recursion
I am trying to explain the value iteration method that is used in reinforcement learning. The method is used to estimate a solution to a recursive equation like:
$ Return(state_t,action_t) = Reward(...
- 178
1
vote
0
answers
24
views
Where can I learn theoretical aspect of Reinforcement learning
Every RL paper has a section with some finite sample analysis/error bound/ convergence proofs. I have difficulty fully understanding such proofs and coming up with my own analysis for my personal ...
- 11
1
vote
0
answers
37
views
Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?
There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
- 1,401
1
vote
0
answers
16
views
Is there any intersection in the applications of multi-agent reinforcement learning and more traditional branches of machine learning?
From my limited understanding, it seems like the structure of problems that multi agent reinforcement learning attempts to attack is quite different from the structure of problems in more traditional ...
- 41
1
vote
0
answers
183
views
Is MCTS an appropriate method for this problem size (large action/state space)?
I'm doing a research on a finite horizon decision problem with $t=1,\dots,40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t)...
- 21
1
vote
0
answers
42
views
Why is the objective in a multi-armed bandit problems to minimize (cumulative) regret?
In a multi-armed bandit, the objective is to maximize the expected cumulative reward. This objective is usually (equivalently?) stated in terms of expected cumulative regret.
Question: Why not just ...
- 371
1
vote
0
answers
18
views
How to compute the sample update error?
In the book Reinforcement Learning An Introduction，Chapter 8.5，there is an example that compares the efficiency of expected and sample updates:
According to the author, "In this case, sample updates ...
- 11
1
vote
0
answers
75
views
What does actually happens on tile hashing
I am going through Richard Sutton book about Reinforcement Learning and I just encountered the tile coding method. I understood pretty well the principles, however, at the very end of the section, ...
- 125
1
vote
0
answers
80
views
Multi Arm Bandit (MAB) -- Increasing reward function
In the general stochastic MAB model, the reward obtained at each trial is generally assumed to be independent of previous trials and obtained from some fixed (but unknown) distribution.
However, if ...
- 175
1
vote
0
answers
31
views
Markov Decision Process Optimal Policy
Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.
Say we have already computed values for the optimal $Q$-...
- 43
1
vote
0
answers
78
views
Policy Gradient Methods
would be really thankful if someone could clear this up.
In reinforcement learning, When we use basis like the fourier or say polynomial basis, and say we have 4 actions, theta will then be a N x4 ...
1
vote
0
answers
122
views
Inconsistent results with Q-learning
I'm running a q-learning agent in tic tac toe against a minimax agent and measuring the tie rate (since you can never win against minimax). When I run 10,000 training games, I've found that there seem ...
- 73
1
vote
0
answers
45
views
Middle ground between model-based and model-free approaches
In the context of the reinforcement learning domain, the dichotomy between model-based (learn a model and used it to determine a controller) and model-free (learn a controller without learning a model)...
- 183
0
votes
0
answers
10
views
RL FOR DYNAMIC ENVIRONMENTS?
I'm new to the field of reinforcement learning (RL), and I’m working on a deep RL project where the environment is constantly changing. In this setup, the state and action spaces not only change based ...
0
votes
0
answers
10
views
Regularized UCB algorithm for multi-armed bandits
I was reading this paper when I stumbled upon what the authors call ``regularized UCB algorithm" (see Appendix A.1). In particular, they define the optimistic mean reward estimator to be
$$\tilde{...
- 101
0
votes
0
answers
22
views
Reinforcement Learning DDPG: creating a meaningful heat map of action value and state
Algorithm: Double Deep Policy Gradient
Output of Actor-Network: Mean action value and variance
Observation (states): x,y coords
My goal is to create a heat map that has a meaningful representation of ...
- 123
0
votes
0
answers
23
views
Calculating sequence of $\lambda$ returns
I am having some trouble with answering the following question:
A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. ...
- 1
0
votes
1
answer
74
views
Main subjects to learn Artificial Intelligence in CS
In my PhD, I will work with ML models. However, I will only use ready-made models as a tool, but I want to delve deeper into Artificial Intelligence not just to use ready-made models, but to ...
- 1
0
votes
0
answers
10
views
Inferring reward function and transition model from optimal policy
Consider an MDP where the transition model and the reward function are unknown.
Consider an optimal policy $\pi^*$ generated from this MDP (say by some oracle who does know the transition model and ...
- 183
0
votes
1
answer
91
views
Motivation for Inverse Reinforcement Learning
Learning a policy from sparse reward information (a reward function where a positive reward is only generated at the goal state) is challenging due to the resulting sparse feedback.
One solution is to ...
- 371
0
votes
0
answers
34
views
How to get best path from a set of path using Q-Learning?
I have 10 data sets (lat and long) of the same path. I started from point A and stopped in point B and did this 10 times for a single route to get the data. While collecting the data, there were some ...
- 1
0
votes
0
answers
127
views
Minimizing the length a Boolean Algebra Expression in disjunctive normal form
I'm looking to minimize the length of an expression in boolean algebra that has been given in disjunctive normal form and is free from redundancy. To remove redundancy from the original expression I ...
0
votes
0
answers
21
views
Path planning agent in a grid with compulsory states to visit with Reinforcement Learning
Is it possible to make a Reinforcement Learning agent for path planning in a 2D grid where visiting certain intermediate states is mandatory?
Please give an insight if it is indeed possible as to ...
- 1
0
votes
0
answers
38
views
Use ML to create a graph
I'm currently looking for literature/papers on machine learning techniques to create structures.
In detail, I want to generate finite automata (NFA, DFA), which are useful for student-exercises. So I ...
0
votes
0
answers
279
views
Converting Turing machine into the source code in industrial programming language?
Are there methods how to convert Turing machine (e.g. neural Turing machine or other rigorous Turing machine) into the source code/program that is written in some industrial programming language like ...
- 1,401
0
votes
0
answers
29
views
Why is bias not defined for non-stationary MDP policies?
The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:
$h(x, \phi) = \sum_{n = 0}^{\infty} E[r(x_n, \phi(x_n)) - w(x_n , \phi)]$
where $x$ is a ...
0
votes
0
answers
134
views
Hessian in reinforcement learning
The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1].
The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues ...
- 11
0
votes
0
answers
53
views
Q-Learning Error Bounds
I have searched a lot for this, but apparently there is no result on calculating any bound on the error $||Q-Q^*||$ when I stop Q-learning after say $N$ iterations ($Q$ is the vector of Q-values at ...
- 175
0
votes
0
answers
321
views
How are punishment and negative reinforcement handled in reinforcement learning using neural net function approximation?
Punishment is reducing a behaviour due to bad outcome e.g. A cow stops touching an electric fence because it gets a shock.
Negative reinforcement is increasing a behaviour that reduces a bad outcome. ...
- 109