SVMs - Fat (Margin) Boundary: Why is $\max\frac1{||\theta||}=\min \frac{ ||\theta||^2}{2}$?

Question

I am trying to understand SVMs in depth watching lectures from MIT.

The professor to reduces the classification problem into an optimization problem. To do that, he first defines the decision and margin boundaries as following:

Note that $\frac1{||\theta||}$ is the distance between the margin boundaries and the decision boundary ($\gamma_i$ is just the signed distance, but we don't care for this question).

We then proceeds to note that our goal is:

• To maximize $\frac1{||\theta||}$
• while minimizing the Hinge Loss

Now, it is apparent that these two terms will co-exist in the objective function we are trying to form $(1)$.

The Objective Function $$\text{minimize } J(\theta,\theta_0) = \frac1n \sum_1^n \text{HingeLoss}(\theta,\theta_0) + \frac{\lambda}{2} ||\theta||^2 \quad (1) $$

Given the fact that he wants to create a minimization problem he describes the maximization of the distance to be equal to the minimization of it's inverse squared divided by 2, i.e: $$\max\frac1{||\theta||}=\min \frac{ ||\theta||^2}{2}$$

This does not make sense, as far as I can understand. In my mind this is the correct way of doing it:

$$\max\frac1{||\theta||}=\min||\theta||$$

Why did he insert a square and a division into the equation?

Answer 1

Minimizing $\|\theta\|$ is equivalent to minimizing $\|\theta\|^2/2$, in the sense that the minimum is achieved for the same value of $\theta$. Since our goal is primarily to find $\theta$, this substitution is not unreasonable.

Why was the substitution done? I don't know. Perhaps because it makes the resulting optimization easier; or just because it seemed simpler. Note that while $\|\theta\|$ might look "simpler", in some sense it is an uglier function, as it involves a square root, which can be messy to deal with. The way to figure out would be to understand the basic derivation using $\|\theta\|^2/2$, and then try replacing with $\|\theta\|$ and see if the same logic goes through or if there is a barrier.