I am reading Bob Carpenter's note at http://lingpipe.files.wordpress.com/2008/04/lazysgdregression.pdf and William Cohen's note at http://www.cs.cmu.edu/~wcohen/10-605/notes/sgd-notes.pdf.

They described the same technique to lazily decay the regularization component of the weight vector (algorithm 2 in Carpenter's). However, I noticed a small difference in their approaches and wondered if this details mattered.

In Carpenter's, the regularization component is updated following $w_i \leftarrow w_i + \eta\frac{u_i - q}{n}\nabla \text{Err}_R$. The vector $\mathbf{u}$ stores the value of $q$ when $u_i$ was last updated. Initially, $\mathbf{u} \leftarrow \mathbf{0}$ and $q=0$. The counter $q$ is incremented after the weight vector has been updated for a particular training sample.

In Cohen's, the regularization component is updated following $w_i \leftarrow w_i \times (1-2\lambda\mu)^{(k-u_i)}$ (by the way, they have different notations for the same thing, both $\lambda$ and $\eta$ refer to the learning rate). The counter $k$ is the same as $q$ in Carpenter's note and incremented before any update to the weight vector. And $\mu$ is the regularization factor.

I have two questions:

  1. Why is Carpenter's adding the regularization component and Cohen's multiplying?

  2. When the first training sample comes in, Carpenter's version essentially ignores the regularization because $u_i - q = 0$. However, Cohen's version does decay the current weight value by a factor of $1-2\lambda\mu$. Does this make any difference?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.