This is more of a conceptual question.
I have learned about Neural Nets, and I have some clue as to how Support Vector Machines work. I read somewhere however that given the appropriate kernel (is that right?), the SVM is identical to the Neural Net. Could someone who understands this please enlighten me as to how that's possible?
An SVM with a linear kernel has the same expressive power as a single perceptron in a neural net, since both are linear classifiers. If you think of the kernel function as mapping your input to a higher dimensional feature space then the SVM is still a linear decision boundary in that high dimensional kernel space. You could theoretically use the high dimensional kernel space features as input to a neural network as well as long as the kernel space is finite. Again the linear SVM behaves similarly to a single perceptron.
However, this only applies to linear SVMs. In practice SVMs are typically used with a non-linear kernel, and then no such equivalence applies. So this claimed equivalence might be a bit misleading, if you are comparing how SVMs tend to be used in practice to how neural nets tend to be used in practice.
Summary: Think of an linear SVM as being similar to a single perceptron instead of a neural network.
For simplicity lets consider a simple single hidden layer feed forward neural net for binary prediction. At test time the neural network predicts
$$ p(Y = 1 \mid X = x) = \sigma(w \cdot \varphi(Ax)), $$ where $w$ is the vector of hidden to output connections, $A$ is the matrix of input to hidden connections, $\sigma$ is the logistic sigmoid function, $\varphi$ is an element wise function (normally something like a sigmoid, tanh, rectified linear, etc), and I've ignored biases for simplicity.
Now skim through the section Nonlinear Classification in the SVM Wikipedia article. Notice how essentially we have the same thing above, but with everything happening in the neural network before the final dot product serving as the transform function $\varphi$, and that's it. The advantage of neural networks is they allow you to easily learn this transform, which can be very beneficial.
some says it is because of this architecture SVM is comparable to neural nets
