0
$\begingroup$

I'm trying to compute the optimization problem for a multi-class SVM loss function with $L2$ regularization.

$\displaystyle f(W) = \frac{1}{n}\sum_{i=1}^n\sum_{c\neq y_i} \max\{0,1-w_{y_i}^Tx_i+w_c^Tx_i\} + \frac{\lambda}{2}||W||^2$ where $W$ is a $k$ by $d$ matrix (data having $k$ classes and $d$ features with $n$ examples)

The link below shows how to compute the gradient and apply to stochastic gradient descent for a usual svm. How do I apply this to a multi-class svm?

https://svivek.com/teaching/lectures/slides/svm/svm-sgd.pdf

My idea was to take the loss function for a random example $i$. Now this loss will be $f_i(W) = \frac{1}{n} \sum_{c\neq y_i}\max\{0,1-w_{y_i}^Tx_i+w_c^Tx_i\}+\frac{\lambda}{2}||W||^2$ and using the fact that the gradient is the sum of all gradients of $f_i(W)$ (i.e., $\nabla f_i(W) = \frac{1}{n}(\sum_{c\neq y_i}\nabla (\max\{0,1-w_{y_i}^Txi+w_c^Tx_i\})+\nabla(\frac{\lambda}{2}||W||^2)$), I will compute the gradient. But I don't know how to compute $\nabla (\max\{0,1-w_{y_i}^Txi+w_c^Tx_i\})$. Can anyone help?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.