1
$\begingroup$

I am reading Bob Carpenter's note at http://lingpipe.files.wordpress.com/2008/04/lazysgdregression.pdf and William Cohen's note at http://www.cs.cmu.edu/~wcohen/10-605/notes/sgd-notes.pdf.

They described the same technique to lazily decay the regularization component of the weight vector (algorithm 2 in Carpenter's). However, I noticed a small difference in their approaches and wondered if this details mattered.

In Carpenter's, the regularization component is updated following $w_i \leftarrow w_i + \eta\frac{u_i - q}{n}\nabla \text{Err}_R$. The vector $\mathbf{u}$ stores the value of $q$ when $u_i$ was last updated. Initially, $\mathbf{u} \leftarrow \mathbf{0}$ and $q=0$. The counter $q$ is incremented after the weight vector has been updated for a particular training sample.

In Cohen's, the regularization component is updated following $w_i \leftarrow w_i \times (1-2\lambda\mu)^{(k-u_i)}$ (by the way, they have different notations for the same thing, both $\lambda$ and $\eta$ refer to the learning rate). The counter $k$ is the same as $q$ in Carpenter's note and incremented before any update to the weight vector. And $\mu$ is the regularization factor.

I have two questions:

  1. Why is Carpenter's adding the regularization component and Cohen's multiplying?

  2. When the first training sample comes in, Carpenter's version essentially ignores the regularization because $u_i - q = 0$. However, Cohen's version does decay the current weight value by a factor of $1-2\lambda\mu$. Does this make any difference?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.