This question refers to this paper: Using Free Energies to Represent Q-values in a Multiagent Reinforcement Learning Task
In section 2.1, equations (5) and (6), I am wondering which Q values are being used to adjust the weights of the restricted boltzmann machine:
Option 1: the Q values generated by the original MDP
Option 2: the approximate Q values obtained by calculating the (negative of the) free energy of the RBM
Follow up question: When considering states $(s^t, a^t)$ and $(s^{t+1}, a^{t+1})$, how do we determine which values of $a^t$ and $a^{t+1}$ to use? Are these the optimal actions from the original MDP, or are these to be obtained through the alternating Gibbs sampling mentioned later on...(which doesn't make sense, since we would not have weights required for this CD)
Thanks for your help