Questions tagged [reinforcement-learning]
The reinforcement-learning tag has no usage guidance.
85
questions
0
votes
1answer
23 views
What is the relation between the compatibile features and the state features in Actor Critic Algorithm?
According to Actor-Critic algorithm, $\psi_{\theta}=\nabla_{\theta}\ln \mu_{\theta}(s, a)$ where $\mu_{\theta}(s, a)$ is the policy followed by the actor and $\psi_\theta$ is the compatibile features ...
0
votes
0answers
8 views
Resource Allocation problem using Deep Reinforcement Learning
I am working on a resource allocation problem, and I need some help.
I have a few Jobs (J) and few Machines (M) to complete them.
Each job has a size (s_j), resource demand (c_j), and a delay bound (...
1
vote
0answers
10 views
Reinforcement learning with 0 rewards and costs
Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
0
votes
0answers
4 views
Inferring reward function and transition model from optimal policy
Consider an MDP where the transition model and the reward function are unknown.
Consider an optimal policy $\pi^*$ generated from this MDP (say by some oracle who does know the transition model and ...
0
votes
1answer
25 views
Motivation for Inverse Reinforcement Learning
Learning a policy from sparse reward information (a reward function where a positive reward is only generated at the goal state) is challenging due to the resulting sparse feedback.
One solution is to ...
0
votes
0answers
20 views
How to get best path from a set of path using Q-Learning?
I have 10 data sets (lat and long) of the same path. I started from point A and stopped in point B and did this 10 times for a single route to get the data. While collecting the data, there were some ...
1
vote
0answers
20 views
Resources on Dynamic Programming with Indefinite Recursion
I am trying to explain the value iteration method that is used in reinforcement learning. The method is used to estimate a solution to a recursive equation like:
$ Return(state_t,action_t) = Reward(...
1
vote
0answers
22 views
Discrete action space for MADDPG
I am trying to apply MADDPG, a policy gradient algorithm that uses centralized training and decentralized execution, to a project. In the work of Lowe et al., the actor returns a pmf for a discrete ...
1
vote
0answers
15 views
Where can I learn theoretical aspect of Reinforcement learning
Every RL paper has a section with some finite sample analysis/error bound/ convergence proofs. I have difficulty fully understanding such proofs and coming up with my own analysis for my personal ...
1
vote
0answers
28 views
Proving facts in inverse reinforcement learning [closed]
I was going through paper titled "Algorithms for Inverse Reinforcement Learning" by Andrew Ng and Russell.
It states following basics:
MDP $M$ is a tuple $(S,A,\{P_{sa}\},\gamma,R)$, where
...
1
vote
1answer
32 views
Reinforcement learining basics - is it possible to deal with environment with random events
As part of data science course I would like to solve particular problem with reinforcement learning algorithm.
I believe I understand general concept, however the problems I had read about up till now ...
1
vote
0answers
38 views
Reinforcement learning and Graph Neural Networks: Issue with entropy [closed]
I am currently working on an experiment to link reinforcement learning with graph neural networks.
This is my architecture:
Feature Extraction with GCN:
there is a fully meshed topology with ...
0
votes
0answers
29 views
Questions on AlphaZero Implementation
So I've been implementing AlphaZero for Chess from scratch and there were a few things the papers mentioned that I'm not sure how to implement. I'll reference both the original AlphaGoZero paper and ...
2
votes
0answers
84 views
Relationship between dynamic programming and reinforcement learning
I wasn't sure whether to post this here or in the ai stack exchange - please let me know if i need to move my post elsewhere)
I have been learning about how dynamic programming can be used as a tool ...
0
votes
1answer
24 views
What is softmax in reinforcement learning?
There are multiple very complicated articles about softmax online. I just want to know a few things about it:
Why do we need softmax? What is the "problem"? What does it do?
How does it do ...
0
votes
0answers
17 views
Is MDP Considered as The Model-based Value Iteration in/of Reinforcement Learning?
Is MDP Considered as The Model-based Value Iteration in/of Reinforcement Learning?
If no, then Reinforcement Learning is all about being Model-free learning. Right?
0
votes
1answer
93 views
Transition Function in MDP
I got a question about who and how sets the transation function values in markov decision processes?
I mean when some says that an agent, in real world grid, is going to step up by %80 and left/right ...
2
votes
1answer
35 views
Policy dependent on initial state distribution in finite horizon MDPs
Consider an MDP defined as the tuple $\langle S,A,R,P,\mu,\lambda\rangle$ where $S$ is the state space, $A$ the action space, $R:S\times A\times S\to\mathbb{R}$ the reward function, $P$ the transition ...
0
votes
0answers
35 views
Is there a difference in the convergence analysis/proof of the chaotic learning automaton compared to the standard LA?
We have recently presented an article entitled Improving learning ability of learning automata using chaos theory.
In this article, a new type of learning automaton called chaotic Learning Automaton (...
0
votes
0answers
19 views
Minimizing the length a Boolean Algebra Expression in disjunctive normal form
I'm looking to minimize the length of an expression in boolean algebra that has been given in disjunctive normal form and is free from redundancy. To remove redundancy from the original expression I ...
1
vote
1answer
35 views
What solution to apply for finding the optimal parameters?
For a study, I have a system (black-box) that requires an input in the form of an array with 4 values (input_array) and depending on their values it produces an ...
1
vote
2answers
33 views
Time efficient way to implement Multi-Armed-Bandits?
I'm doing a research on Multi-Armed Bandit (MAB) problem with approx. 1 million arms. In contrast, the number of iterations is of course much larger, about 10-20 million.
Most MAB-algorithms require ...
0
votes
0answers
17 views
Path planning agent in a grid with compulsory states to visit with Reinforcement Learning
Is it possible to make a Reinforcement Learning agent for path planning in a 2D grid where visiting certain intermediate states is mandatory?
Please give an insight if it is indeed possible as to ...
1
vote
1answer
42 views
What are the basics of CS i should know,before I start my journey into machine learning
I am myself a non-cs graduate and would love to be a machine learning engineer.
I have learned to code and know the basics of <...
0
votes
0answers
29 views
Use ML to create a graph
I'm currently looking for literature/papers on machine learning techniques to create structures.
In detail, I want to generate finite automata (NFA, DFA), which are useful for student-exercises. So I ...
0
votes
0answers
137 views
Converting Turing machine into the source code in industrial programming language?
Are there methods how to convert Turing machine (e.g. neural Turing machine or other rigorous Turing machine) into the source code/program that is written in some industrial programming language like ...
1
vote
0answers
23 views
Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?
There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
1
vote
0answers
7 views
Is there any intersection in the applications of multi-agent reinforcement learning and more traditional branches of machine learning?
From my limited understanding, it seems like the structure of problems that multi agent reinforcement learning attempts to attack is quite different from the structure of problems in more traditional ...
0
votes
0answers
29 views
Why is bias not defined for non-stationary MDP policies?
The Handbook of Markov Decision Processes 1 defines bias of a stationary deterministic policy as follows:
$h(x, \phi) = \sum_{n = 0}^{\infty} E[r(x_n, \phi(x_n)) - w(x_n , \phi)]$
where $x$ is a ...
1
vote
0answers
158 views
Is MCTS an appropriate method for this problem size (large action/state space)?
I'm doing a research on a finite horizon decision problem with $t=1,\dots,40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t)...
1
vote
0answers
30 views
Why is the objective in a multi-armed bandit problems to minimize (cumulative) regret?
In a multi-armed bandit, the objective is to maximize the expected cumulative reward. This objective is usually (equivalently?) stated in terms of expected cumulative regret.
Question: Why not just ...
1
vote
0answers
17 views
How to compute the sample update error?
In the book Reinforcement Learning An Introduction，Chapter 8.5，there is an example that compares the efficiency of expected and sample updates:
According to the author, "In this case, sample updates ...
1
vote
1answer
167 views
Help needed for understanding proof of No Regret Multi Armed Bandit Algorithm
I was reading Elad Hazan's book on Online Convex Optimization(http://ocobook.cs.princeton.edu/OCObook.pdf) and am facing difficulty understanding the proof given for the No regret algorithm for MAB (...
1
vote
0answers
42 views
What does actually happens on tile hashing
I am going through Richard Sutton book about Reinforcement Learning and I just encountered the tile coding method. I understood pretty well the principles, however, at the very end of the section, ...
0
votes
1answer
355 views
Understanding On-policy First Visit Monte Carlo Control algorithm
I am going through the Monte Carlo methods, and it's going fine until now. However, I am actually studying the On-Policy First Visit Monte Carlo control for epsilon soft policies, which allows us to ...
3
votes
1answer
123 views
(DROP) Data Reduction Algorithm - How it works?
I am studing a PHD framework which the propose is to reduce the dataset with the most representative samples for training a classifier. Maybe I am loosing something, but I could not undestand a ...
0
votes
2answers
91 views
Is the credit assignment problem a well-posed one?
Credit assignment is the process of assigning credit (or blame) to a particular move in a sequence of moves (temporal credit assignment) or to a particular node (structural credit assignment) among ...
1
vote
1answer
404 views
Is deep learning appropriate to approximate dynamic programming problems?
I have a problem which can be completely solved using dynamic programming, but in a very intractable way (On^4, where n is around 1000). I won't get into the details of the problem since it's a bit ...
0
votes
0answers
86 views
Hessian in reinforcement learning
The Hessian of multi-layered network exhibits known behaviour at critical points as shown in [1].
The tools of random matrix theory allow [2] to deduce the asymptotic distribution of the eigenvalues ...
0
votes
1answer
165 views
Existence of Optimal Policy for infinite-state MDPs
It is well-know from Puterman's book (1994) that in any finite-state MDP, if there exists an optimal policy, then that policy is stationary and deterministic. How about MDPs with continuous state ...
1
vote
0answers
62 views
Multi Arm Bandit (MAB) — Increasing reward function
In the general stochastic MAB model, the reward obtained at each trial is generally assumed to be independent of previous trials and obtained from some fixed (but unknown) distribution.
However, if ...
0
votes
0answers
43 views
Q-Learning Error Bounds
I have searched a lot for this, but apparently there is no result on calculating any bound on the error $||Q-Q^*||$ when I stop Q-learning after say $N$ iterations ($Q$ is the vector of Q-values at ...
1
vote
0answers
29 views
Markov Decision Process Optimal Policy
Consider the setting of finite MDPs. I will be using the notation in Chapter 2 of http://rll.berkeley.edu/deeprlcourse/docs/ng-thesis.pdf.
Say we have already computed values for the optimal $Q$-...
0
votes
2answers
118 views
Eligibility trace and the role of gamma and lambda
From R.Sutton's book, the eligibility trace update rule is:
$$
E_t(s)\leftarrow\gamma~\lambda~ E_{t-1}(s)+\mathbb{1}(S_t=s)
$$
I wonder why do we need both $\gamma$ and $\lambda$ to assign credit to ...
1
vote
1answer
108 views
Inverse Reinforcement Learning and Apprenticeship Learning differences
IRL as I understand is where the AI is given a policy from an "expert" and needs to solve for the reward function which best fits the policy.
And Apprenticeship Learning via IRL was explained as an ...
3
votes
1answer
1k views
Approximate value iteration for continuous state space MDPs
I have a continuous state space MDP as a generative model. I input the state and action and it outputs the reward and the next state. Assume that I sampled $n$ state-reward-states. I wonder how I can ...
1
vote
1answer
183 views
Exploration in reinforcement learning when state space is huge
When the state-action space is huge I know I can use function approximators to generalize but how can I explore? Doing an exhaustive search seems very naive. What DeepMind for example did to explore ...
5
votes
2answers
240 views
Why does the effectiveness of my reinforcement based neural network recede after a while?
I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article.
Typically, ...
1
vote
1answer
81 views
Bayesian updating for multivariate Gaussian
I am reading http://www.yisongyue.com/courses/cs159/lectures/LinUCB.pdf and come across this slide
What has been confusing me boils down to showing that multivariate Gaussian is conjugate to itself ...
6
votes
1answer
533 views
Machine learning approach to auction game
I am newbie with machine learning. In order to learn more I decided to try solving a specific problem/game that I have in mind. The problem is the following:
I have a list of $N$ items which are ...
