Questions tagged [reinforcement-learning]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
10
votes
1answer
8k views

The meaning of discount factor on reinforcement learning

After reading of the google deepmind achievements on Atari's games, I am trying to understand the q-learning and q-networks, but I am little bit confused. The confusion arise in the concept of the ...
6
votes
1answer
829 views

What's the difference between Adaptive Control and Hierarchical Reinforcement Learning?

After watching Travis DeWolf presentation on scaling neural computation, I'm a bit confused about the difference between Reinforcement Learning (whether hierarchical or not) and Adaptive Control. They ...
6
votes
1answer
532 views

Machine learning approach to auction game

I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following: I have a list of $N$ items which are ...
5
votes
1answer
3k views

What is the difference between apprenticeship and imitation learning?

There are many sub-domains of reinforcement learning, two of which are apprenticeship and imitation learning - but that appears just to be two different names for the same thing?
5
votes
1answer
832 views

Dyna-Q in non-deterministic domains

I've implemented the Dyna-Q reinforcement learning algorithm and it works perfectly on a discrete deterministic environment, the cliff. However, when applying it to a continuous environment (mountain ...
5
votes
1answer
151 views

Reinforcement Learning - Agent training

In RL, in a game situation, usually the agent is trained by playing against itself. When we should not depend on this self-training, and switch to train the agent with a real or different player ...
5
votes
2answers
238 views

Why does the effectiveness of my reinforcement based neural network recede after a while?

I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article. Typically, ...
4
votes
1answer
46 views

When do the gradients exist in a Reinforcement Learning setting?

I am getting properly stuck into reinforcement learning and I am currently reading the review paper by Kober et al. (2013). And there is one constant feature that I cannot get my head around, but ...
4
votes
1answer
276 views

reinforcement learning in gridworld with subgoals

Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. There is a specific example ...
3
votes
2answers
4k views

Q-learning in a Dynamic environment

I am new to reinforcement learning. Lately, I have learned Q-learning using the following tutorial. Is Q-learning still possible if the environment is dynamic. Using the environment of the tutorial ...
3
votes
2answers
2k views

Reinforcement Learning: An Introduction, A Gambler's Problem, Exercise 4.7 Solution

Currently reading a recent draft of Reinforcement Learning: An Introduction by Sutton and Barto. Really good book! I was a bit confused by exercise 4.7 in chapter 4, section 4, page 93, (see ...
3
votes
1answer
397 views

Why does ε-greedy $Q$-learning not oscillate?

I have a intuitive question on the convergence of $Q$- learning. In $Q$ learning for each step a $Q$- value is learned for the state-action pair where the action is selected according to the $\epsilon$...
3
votes
1answer
2k views

Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?

I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except ...
3
votes
1answer
1k views

Approximate value iteration for continuous state space MDPs

I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can ...
3
votes
1answer
41 views

Can I have actions that, if taken, cannot be taken again in the following N timesteps?

I'm trying to train a load-balancing system with reinforcement-learning (RL) s.t. the incoming jobs are queued evenly at the available servers. The system will not be able to directly dispatch the ...
3
votes
1answer
114 views

(DROP) Data Reduction Algorithm - How it works?

I am studing a PHD framework which the propose is to reduce the dataset with the most representative samples for training a classifier. Maybe I am loosing something, but I could not undestand a ...
3
votes
1answer
1k views

How is the environment designed for testing a reinforcement learning algorithm?

I'm working on a project, and have a candidate algorithm which I'd like to test. Before I go any further, I need to get the hang of how to code the "structure" of the environment in which my system is ...
3
votes
0answers
528 views

Reinforcement learning - state space and action space

I am working on a reinforcement learning strategy for parameter control of a local search heuristic. The complete state for this RL problem can be defined as $S = \{s, p\}$, where $s$ and $p$ ...
2
votes
1answer
683 views

Why do we use the log in gradient-based reinforcement algorithms?

I've been reading some papers on reinforcement learning. $$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$ I often see expressions, similar to the above one, where the weights (denoted by $w$) are ...
2
votes
2answers
150 views

Reinforcement learning without self-play

Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)? For example, let'...
2
votes
1answer
77 views

Some questions on kernels and Reinforcement Learning

I've a test in a few days and I've a few issues with some of the subjects. Let's start with kernels, basically I understood that a kernel needs to be positive semi-definite and symmetric in order to ...
2
votes
1answer
159 views

Reference request: Introduction to reinforcement learning with hand calculation examples

For me, the most difficulty when it comes to learning about reinforcement learning is that there is not much to learn in the sense that without running some algorithm, it is very difficult to get a ...
2
votes
1answer
151 views

Reinforcement Learning - Q Learning

I am having trouble understanding the following problem and Q learning in general. What I know so far about Q learning is that Q-learning is a model free method, i.e., it doesn’t need to learn P(s’|...
2
votes
1answer
1k views

single agent vs multiple agent reinforcement learning

I am confused about 'single' vs 'multiple' agent reinforcement learning. Let's say that I have 1 hunter who I am training to hunt 1 static prey, so that only the hunter is moving around. This is ...
2
votes
1answer
171 views

How to determine convergence when using Q-learning?

I'm using Q-Learning to find the values of states in on a gameboard. For example, something like: ...
2
votes
1answer
353 views

How does SARSA handle episode termination

When applied to domains that are episodic and have a "final" state but no final action, like a game, how does SARSA update the Q-values? e.g. A game agent would receive this series: $$ s_0,a_0,...
2
votes
0answers
64 views

Relationship between dynamic programming and reinforcement learning

I wasn't sure whether to post this here or in the ai stack exchange - please let me know if i need to move my post elsewhere) I have been learning about how dynamic programming can be used as a tool ...
2
votes
1answer
30 views

Policy dependent on initial state distribution in finite horizon MDPs

Consider an MDP defined as the tuple $\langle S,A,R,P,\mu,\lambda\rangle$ where $S$ is the state space, $A$ the action space, $R:S\times A\times S\to\mathbb{R}$ the reward function, $P$ the transition ...
2
votes
0answers
127 views

Q-learned policy differs from double-Q-learned policy

I implemented Q-learning and double-Q-learning as presented in Sutton's "Reinforcement Learning: An Introduction". I test the algorithms on the OpenAI cliff walking gym and analyze the resulting ...
2
votes
0answers
66 views

How to best model multidimensional, continuous, non-convex “shape” as neural network?

I have: set of n-dimensional points that I know are inside of the shape n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity). shape is pretty large in this n-...
1
vote
1answer
4k views

What does it mean to have a continuous action space w.r.t. to reinforcement learning?

Last time I posted this question I got criticised for not being specific enough, hence this is my second attempt at trying to understand what it means to have a continuous action space. Please refer ...
1
vote
1answer
29 views

Reinforcement learining basics - is it possible to deal with environment with random events

As part of data science course I would like to solve particular problem with reinforcement learning algorithm. I believe I understand general concept, however the problems I had read about up till now ...
1
vote
1answer
42 views

What are the basics of CS i should know,before I start my journey into machine learning

I am myself a non-cs graduate and would love to be a machine learning engineer. I have learned to code and know the basics of <...
1
vote
1answer
177 views

Exploration in reinforcement learning when state space is huge

When the state-action space is huge I know I can use function approximators to generalize but how can I explore? Doing an exhaustive search seems very naive. What DeepMind for example did to explore ...
1
vote
1answer
35 views

What solution to apply for finding the optimal parameters?

For a study, I have a system (black-box) that requires an input in the form of an array with 4 values (input_array) and depending on their values it produces an ...
1
vote
2answers
33 views

Time efficient way to implement Multi-Armed-Bandits?

I'm doing a research on Multi-Armed Bandit (MAB) problem with approx. 1 million arms. In contrast, the number of iterations is of course much larger, about 10-20 million. Most MAB-algorithms require ...
1
vote
1answer
165 views

Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm

I was reading Elad Hazan's book on Online Convex Optimization(http://ocobook.cs.princeton.edu/OCObook.pdf) and am facing difficulty understanding the proof given for the No regret algorithm for MAB (...
1
vote
1answer
398 views

Is deep learning appropriate to approximate dynamic programming problems?

I have a problem which can be completely solved using dynamic programming, but in a very intractable way (On^4, where n is around 1000). I won't get into the details of the problem since it's a bit ...
1
vote
1answer
106 views

Inverse Reinforcement Learning and Apprenticeship Learning differences

IRL as I understand is where the AI is given a policy from an "expert" and needs to solve for the reward function which best fits the policy. And Apprenticeship Learning via IRL was explained as an ...
1
vote
1answer
1k views

Difference between SNN RL and DNN RL?

In Reinfrocement Learning (RL) in Neural Networks (NNs), I've seen two approaches to Q-learning. The first is to tile the state space with basis functions using Spiking Neural Networks (SNN) to ...
1
vote
1answer
286 views

Using a combination of spatial and non-spatial inputs for convolutional neural networks

I'm working on training a game AI using deep reinforcement learning to achieve specific examples based on pixel input and some additional state information. Naturally, I'm using a convolutional ...
1
vote
1answer
72 views

Combining Production Rules using Reinforcement Learning

Production systems have been used to solve puzzles such as the Tower of Hanoi for years with hard-coded production rules. However, has there been any research in using reinforcement learning to ...
1
vote
0answers
10 views

Reinforcement learning with 0 rewards and costs

Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
1
vote
0answers
20 views

Resources on Dynamic Programming with Indefinite Recursion

I am trying to explain the value iteration method that is used in reinforcement learning. The method is used to estimate a solution to a recursive equation like: $ Return(state_t,action_t) = Reward(...
1
vote
0answers
22 views

Discrete action space for MADDPG

I am trying to apply MADDPG, a policy gradient algorithm that uses centralized training and decentralized execution, to a project. In the work of Lowe et al., the actor returns a pmf for a discrete ...
1
vote
0answers
15 views

Where can I learn theoretical aspect of Reinforcement learning

Every RL paper has a section with some finite sample analysis/error bound/ convergence proofs. I have difficulty fully understanding such proofs and coming up with my own analysis for my personal ...
1
vote
0answers
28 views

Proving facts in inverse reinforcement learning [closed]

I was going through paper titled "Algorithms for Inverse Reinforcement Learning" by Andrew Ng and Russell. It states following basics: MDP $M$ is a tuple $(S,A,\{P_{sa}\},\gamma,R)$, where ...
1
vote
0answers
37 views

Reinforcement learning and Graph Neural Networks: Issue with entropy [closed]

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture: Feature Extraction with GCN: there is a fully meshed topology with ...
1
vote
0answers
23 views

Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?

There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
1
vote
0answers
7 views

Is there any intersection in the applications of multi-agent reinforcement learning and more traditional branches of machine learning?

From my limited understanding, it seems like the structure of problems that multi agent reinforcement learning attempts to attack is quite different from the structure of problems in more traditional ...