Support Vector Machines as Neural Nets?

This is more of a conceptual question.

I have learned about Neural Nets, and I have some clue as to how Support Vector Machines work. I read somewhere however that given the appropriate kernel (is that right?), the SVM is identical to the Neural Net. Could someone who understands this please enlighten me as to how that's possible?

• Where did you read that, and what exactly did the source say? Without a specific reference and a more precise quotation, it's harder to comment on that assertion. – D.W. Sep 26 '14 at 0:59

An SVM with a linear kernel has the same expressive power as a single perceptron in a neural net, since both are linear classifiers. If you think of the kernel function as mapping your input to a higher dimensional feature space then the SVM is still a linear decision boundary in that high dimensional kernel space. You could theoretically use the high dimensional kernel space features as input to a neural network as well as long as the kernel space is finite. Again the linear SVM behaves similarly to a single perceptron.

However, this only applies to linear SVMs. In practice SVMs are typically used with a non-linear kernel, and then no such equivalence applies. So this claimed equivalence might be a bit misleading, if you are comparing how SVMs tend to be used in practice to how neural nets tend to be used in practice.

Summary: Think of an linear SVM as being similar to a single perceptron instead of a neural network.

• I don't think the kernel function maps input into higher dimensional space. What the kernel function does is mapping input into another space that will be homeomorphic to the original one – InformedA Sep 26 '14 at 4:10
• I believe you are correct that the kernel function doesn't need to map into a higher dimensional space. However, in practice the popular kernel functions (polynomial kernels, rbf kernel, ...) do map into higher dimensions. This is the whole purpose of the kernel trick. – Aaron Sep 26 '14 at 4:17

For simplicity lets consider a simple single hidden layer feed forward neural net for binary prediction. At test time the neural network predicts

$$p(Y = 1 \mid X = x) = \sigma(w \cdot \varphi(Ax)),$$ where $w$ is the vector of hidden to output connections, $A$ is the matrix of input to hidden connections, $\sigma$ is the logistic sigmoid function, $\varphi$ is an element wise function (normally something like a sigmoid, tanh, rectified linear, etc), and I've ignored biases for simplicity.

Now skim through the section Nonlinear Classification in the SVM Wikipedia article. Notice how essentially we have the same thing above, but with everything happening in the neural network before the final dot product serving as the transform function $\varphi$, and that's it. The advantage of neural networks is they allow you to easily learn this transform, which can be very beneficial.

some says it is because of this architecture SVM is comparable to neural nets