I'm trying to compute the optimization problem for a multi-class SVM loss function with $L2$ regularization.

$\displaystyle f(W) = \frac{1}{n}\sum_{i=1}^n\sum_{c\neq y_i} \max\{0,1-w_{y_i}^Tx_i+w_c^Tx_i\} + \frac{\lambda}{2}||W||^2$ where $W$ is a $k$ by $d$ matrix (data having $k$ classes and $d$ features with $n$ examples)

The link below shows how to compute the gradient and apply to stochastic gradient descent for a usual svm. How do I apply this to a multi-class svm?


My idea was to take the loss function for a random example $i$. Now this loss will be $f_i(W) = \frac{1}{n} \sum_{c\neq y_i}\max\{0,1-w_{y_i}^Tx_i+w_c^Tx_i\}+\frac{\lambda}{2}||W||^2$ and using the fact that the gradient is the sum of all gradients of $f_i(W)$ (i.e., $\nabla f_i(W) = \frac{1}{n}(\sum_{c\neq y_i}\nabla (\max\{0,1-w_{y_i}^Txi+w_c^Tx_i\})+\nabla(\frac{\lambda}{2}||W||^2)$), I will compute the gradient. But I don't know how to compute $\nabla (\max\{0,1-w_{y_i}^Txi+w_c^Tx_i\})$. Can anyone help?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.