I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except using maxQ at s' instead of the actual a', as required by Q-learning; I've solved it with other methods easily, the question is about linear Q-learning). Also, I'm using 1 output for each action, so I can select an action in a single forward pass. Important: my input features are the identity function of the state, i.e., the state variables themselves. I know it can be solved with other features, I want to know about this one specifically.
Here is a description of the algorithm I'm using: https://youtu.be/vVDKzIxzkzQ
Unfortunately, this method never solves the task. Actually, it rarely reaches the top. I'm aware that vanilla Q-learning has no convergence guarantees, so I tried some variants which reduce or even solve this issue: I've tried experience replay, double q-learning, GTD and advantage learning. Nothing helped.
I know that this task can be solved linearly on the state inputs directly, since I've done it by manually tuning the weights so the car always apply the force in the same direction as its velocity, so the representation is not an issue. But it seems that the learning algorithm is.
So the question is: is linear Q-learning able to learn the mountain car task directly from the state variables? A paper where the author solves this task with Q-learning in a single-layer network (and shows the algorithm or code) with features = state variables would be enough.
