3
$\begingroup$

I am new to reinforcement learning. Lately, I have learned Q-learning using the following tutorial.

Is Q-learning still possible if the environment is dynamic. Using the environment of the tutorial as an example, in some states it is possible to go from room 0 to room 1 and and it is not possible to go from room 0 to 4.

Is it possible to use Q-learning in a problem like that? if not is there an algorithm that handle dynamic environments?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

In general you can't change the state and action spaces (rows and columns of your reward matrix) since that changes the underlying model that you are sampling with Q-learning. It'd be similar to changing the population you're sampling when statistically sampling. Hence, you have to define a set of actions and states that don't change in order to use Q-learning. Instead of the house graph, you'd need something else. Ideally you'd have to come up with a set of states and actions that didn't change as the house's topology changed. E.g. actions could be strategy to use could be the action given the number of rooms in the house and total number of edges (note: this'd probably be a poor choice, but I needed an example.).

All that said Q-learning albeit a little slow is pretty robust to some types of disturbances. If the model doesn't change significantly (e.g. you add a room) and you are exploring a large state space (statistically), then your older policies (solved Q matrices) might be nice breadcrumb trails for your new algorithm's exploration strategy, and the algorithm may re-learn much faster. However, I don't know about the theoretical guarantees (look into terms like "cross learning").

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the clear answer. How about if the rewards change over time for example: sometimes to move from 1 to 5, you have to pay 100 (so in that case it is better to go 1,3,4,5 because you don't have to pay anything). Sometimes, you can go from 1 to 5 for free (so in this case you go from 1 to 5 instead 1,3,4,5). In this case it is possible to use Q-learning ? if not is there a better algorithm that can handle such a case ? $\endgroup$
    – user655561
    May 14, 2014 at 19:10
  • 2
    $\begingroup$ In many cases in Q-learning rewards are stochastic. Q-learning will be fine with this. Think of the reward value in the table as being an average reward. For instance if you randomly changed the cost between 50 and 100 each time you took a link, your system observes an average cost of 75. Run it enough, and it should work out. $\endgroup$
    – Tim
    May 14, 2014 at 23:38
  • 1
    $\begingroup$ BTW: my favorite RL text is webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html $\endgroup$
    – Tim
    May 14, 2014 at 23:40
-1
$\begingroup$

You should check this paper: https://ieeexplore.ieee.org/document/7849368 It discusses Q-Learning in a dynamic environment.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ This paper seems relevant. However, we expect answers to stand on their own, and not depend on (in this case possibly paywalled?) external links. Could you improve your answer by adding a brief summary of the article, or explain how it applies to the example mentioned in the question? Thanks! $\endgroup$
    – Discrete lizard ♦
    Jun 19, 2019 at 5:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.