What is the relation between the compatibile features and the state features in Actor Critic Algorithm?

Question

According to Actor-Critic algorithm, $\psi_{\theta}=\nabla_{\theta}\ln \mu_{\theta}(s, a)$ where $\mu_{\theta}(s, a)$ is the policy followed by the actor and $\psi_\theta$ is the compatibile features space.

The updates for the actor and critic parameters are as follows:

For critic with the parameter w:

$w_{t+1} = w_t + \beta_t\delta_t^{\mu_{\theta}}\phi(s_t)$

where $\phi(s_t)$ is the feature vector of state $s_t$

For actor with the parameter theta:

$\theta_{t+1}=\theta_t + \alpha_t\delta_t^{\mu_{\theta}}\psi(s_t, a_t)$

where $\psi(s_t, a_t)$ is the compatible feature vector of state $s_t$, with action $a_t$.

I am trying to understand the relation between $\psi(s_t, a_t)$ and $\phi(s_t)$, any pointers would be appreciated.

Answer 1

$\phi(s)$ is the representation of the state, and $\psi$ is the gradient (of the log) of the policy. Essentially, those two don't really have any connection, except for that the input to the policy is the state's representation. That is, the policy will compute the appropriate action distribution according by using its model (say a neural network) when applied on the representation of a state.