Let's say we have trained a Support Vector Machine with a Gaussian Kernel. When we feed our model an example, it classifies it based on its similarity to landmarks (distance to examples in our training set).

If instead our model is a K-Nearest Neighbors algorithm, with k=size of the training set, it classifies a given example based on its distance to examples in our training set (similarity to landmarks).

I know the math behind SVM and KNN is different, but on a high level, are they both employing the same idea? What am I missing if I think they are doing the same thing in different ways?

  • 1
    $\begingroup$ Using K-NN while letting $k$ be the size of the training set yields a constant function (the majority vote in your training set), which means it's a bad idea. $\endgroup$ – Ariel Sep 1 '16 at 14:38
  • 1
    $\begingroup$ SVM doesn't classifies based on the distance to examples. SVM gives hyper plane that separates the different labels. The classification is with regard to the learned hyper plane and not the examples in training set $\endgroup$ – Amitay Nachmani Sep 1 '16 at 18:24
  • 1
    $\begingroup$ "k=size of the training set" is a typo correct? Did you mean "k=number of classes"? $\endgroup$ – William Schaller Sep 21 '16 at 21:51
  • $\begingroup$ Both, as in k = size of the training set = number of classes $\endgroup$ – Atte Juvonen Sep 22 '16 at 4:11
  • $\begingroup$ Interesting. Typically datasets contains hundreds if not thousands of examples of each class. Not that this changes the question, though. $\endgroup$ – William Schaller Sep 23 '16 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.