Support Vector Machines vs K-Nearest Neighbors

Let's say we have trained a Support Vector Machine with a Gaussian Kernel. When we feed our model an example, it classifies it based on its similarity to landmarks (distance to examples in our training set).

If instead our model is a K-Nearest Neighbors algorithm, with k=size of the training set, it classifies a given example based on its distance to examples in our training set (similarity to landmarks).

I know the math behind SVM and KNN is different, but on a high level, are they both employing the same idea? What am I missing if I think they are doing the same thing in different ways?

• Using K-NN while letting $k$ be the size of the training set yields a constant function (the majority vote in your training set), which means it's a bad idea. Sep 1, 2016 at 14:38
• SVM doesn't classifies based on the distance to examples. SVM gives hyper plane that separates the different labels. The classification is with regard to the learned hyper plane and not the examples in training set Sep 1, 2016 at 18:24
• "k=size of the training set" is a typo correct? Did you mean "k=number of classes"?
– Wil
Sep 21, 2016 at 21:51
• Both, as in k = size of the training set = number of classes Sep 22, 2016 at 4:11
• Interesting. Typically datasets contains hundreds if not thousands of examples of each class. Not that this changes the question, though.
– Wil
Sep 23, 2016 at 21:47

Obviously, classification rests on the idea of similarity. Always. And in a metric space, similarity is measured with distances, and equal-distance loci are planes. [In fact, you try to minimize the intra-class distances while maximizing the inter-class distances.]

SVM:

• Faster to compute
• Requires less data to validate
• Less accurate than KNN where KNN is for nonlinear data as well.

Hint: Use a kernel with SVM, then you will get a very good result.

• I don't see how this answers the question. The question was "on a high level, are they both employing the same idea?" and "What am I missing if I think they are doing the same thing in different ways?" I don't see an answer to either of those. If you see this as answering the question, I encourage you to edit your post to more explicitly state your answer to each of those questions.
– D.W.
Sep 11, 2023 at 16:50