3
$\begingroup$

I'm wondering if choosing K in the Nearest Neighbors Classifier can be stated as model selection, since the computational complexity is getting higher, the higher K is.

My understanding is: since the complexity increases it means, that K is a hyper parameter.

Am I getting the concept of hyper parameters correctly?

$\endgroup$
1
$\begingroup$

First let's define what a hyper parameter is. In the context of statistical learning, a hyper parameter is a fixed constant that describes the prior distribution for a model's parameters. For example, if $X \sim \text{Binomial}(n,p)$ and $p \sim \text{Beta}(\alpha, \beta)$, then $\alpha, \beta$ would be hyper parameters where $n, p$ would be the "parameters" (I use quotes because $p$ is a random variable now).

$k$NN is non-parametric in the sense that you aren't explicitly modeling your data as a function of underlying parameters. $k$ only describes how many neighbors to "learn" from. As $k$ increases the computational complexity can increase; however, it doesn't explicitly control the complexity of the model. An example where model size $k$ acts as a hyper parameter is linear regression. Imagine trying to determine a-priori which $k$ of the $N$ features you would like to select for inclusion in the model.

Given this, I would not say that $k$ describes $k$NN model complexity in the sense that typical Bayesian model selection is and therefore should not be viewed as a hyperparameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.