One of the advantage of using a features reduction/sub feature selection is to avoid a high dimentionality.

The most known method is the Forward selection, where it finds the best combinations of features among others, iteratively. Now please have a look at this picture retrieved from a ppt slide on the internet.

enter image description here

The second row of the table shows that the 11 features are the features selected/reduced from initial 20 features. My question is that, the 11 features are still considered as a high-dimensional features set, indeed one can't just subtitute it over a KNN function. Now I'm confused as to how he/she then adapt it to the KNN classifier for the 11 features?

  • 1
    $\begingroup$ KNN could be applied to any number of dimension. You just need a distance function between two given samples. Why do you think it could not be done for 11 features? $\endgroup$ – Apiwat Chantawibul Sep 1 '17 at 22:30
  • $\begingroup$ Hi Bulliska. I mean, you'll encounter so called the curse of dimentionality, where using many features in 1 feature vector can cause a classification accuracy produces in a low rate value. Usually people prefer euclidean in 2 or 3D. Not higher than it even though we can just use the many features directly. $\endgroup$ – Plain_Dude_Sleeping_Alone Sep 2 '17 at 5:55
  • 1
    $\begingroup$ Thanks for your clarification. I'm just trying to help make your question answerable. What number of dimensions is not suitable clearly depends on data. Imagine a data set in $ n $ dimension where the two classes cluster perfectly at $0^n $ and $1^n $ clearly what $ n $ is you can still use KNN $\endgroup$ – Apiwat Chantawibul Sep 2 '17 at 9:39
  • $\begingroup$ And so it would seem your question might not be about the application of KNN in general, but rather the application of KNN in the particular experiment. Have I summarise this correctly? $\endgroup$ – Apiwat Chantawibul Sep 2 '17 at 9:44
  • $\begingroup$ I must admit that the question is rather vague. But I think is more like, how do I go process these features into KNN without encountering the high dimentionality. Perhaps in the case of the paper, it went smoothly because the 11 features are strongly correlated to one another. So maybe I just need an explanation in general that the number of features used are still relevant in the paper. Thank you for your time responding, I really appreciate it, it's an honour. $\endgroup$ – Plain_Dude_Sleeping_Alone Sep 2 '17 at 16:35

There seems to be a misconception. You can use $k$-nearest neighbors with any number of dimensions. There is no prohibition on using it with a large number of features. And in some cases $k-NN will work well even in high dimensions (a large number of features); it all depends on the dataset, and to find out whether it will work well, you have to try it. In general, feature selection might make the algorithm work better, or it might make it work worse -- it all depends.

| cite | improve this answer | |
  • $\begingroup$ Hi @DW Thank you for the answer and the time taken for it, though, I didn't consider the case as prohibition, but rather tendention to avoid such. However I accept and agree to your answer :). $\endgroup$ – Plain_Dude_Sleeping_Alone Sep 7 '17 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.