I've been reading some papers on reinforcement learning.

$$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$

I often see expressions, similar to the above one, where the weights (denoted by $w$) are updated following the partial derivative of the policy function (denoted by $p_w$) with respect to its weights.

But why do we take the $\log$? What is its purpose?


We often take the logarithm because:

  1. Maximizing $\log \Phi(x)$ is equivalent to maximizing $\Phi(x)$, so in maximum-likelihood problems, we can maximize the log of the likelihood instead of maximizing the likelihood directly and the result will be equivalent.

  2. The logarithm converts multiplication to addition, and the derivative of a sum is "nicer" than the derivative of a product. Products often arise when maximizing likelihoods, because the probability of something can often be written as the product of other probabilities (e.g., when we are calculating the probability of that multiple independent events all happen). The derivative of the product is ugly, but the derivative of the log of the product is nice and simple.

  3. Gradient descent often involves maximizing a likelihood, so we'll need to take derivatives. The expressions get simpler/cleaner if we can take the derivative of the log likelihood instead.

| cite | improve this answer | |
  • $\begingroup$ I think you should have been more specific to this context. You should have answered to the questions: 1) How is $\pi_w$ is usually represented and thus derived? 2) Why does $\ln \pi_w$ is a simpler to derive than $\pi_w$? $\endgroup$ – nbro Feb 19 '19 at 10:41
  • $\begingroup$ @nbro, those weren't the questions that were asked. If you'd like to write your own answer based on that, feel free. $\endgroup$ – D.W. Feb 19 '19 at 17:00
  • $\begingroup$ The question was asked in the context of RL, not in the general case, so those questions, even though not explicit, should have been answered. But if the OP accepted your answer, I think he got satisfied. $\endgroup$ – nbro Feb 19 '19 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.