2
$\begingroup$

Having a unsupervised algorithm done on a dataset (e.g. K-means), how can I determine which cluster is normal and which is anomalous (for 30 clusters, for example)?

If the dataset contains normal traffic much more than anomalies (99%), then the radius of clusters associate to normal traffic is bigger. Is this true? Why?

If so, then we set the clusters with greater radius to normal according to a threshold. The smaller the threshold, the more the detection rate and the more the false alarm rate. Is this true?

Could a labeled dataset help determine the normal clusters? For example, finding the closest point to the centroid of a given cluster and checking whether it is labeled anomaly or not?

[1] uses Cluto software and its mountain visualization graph that I can't understand why its 3D, having feature space of dimension 41.

Reference: [1] Jose F. Nieves, Data Clustering for Anomaly Detection in Network Intrusion Detection

$\endgroup$
1
$\begingroup$

The radius depends on the number of clusers you choose. The more clusters, the smaller the radius.

If you have labelled data, you could eliminate the outliers/anomalies for determining the threshold radii for your clusters. For example you could take the greatest distance from the cluster center to its members (smaller radius for higher detection rate as stated in your question).

This question is closely related to the question Anomaly/outlier detection using fuzzy clustering. I am reposting my answer:

As far as I know there is no unique definition for what is an outlier/anomaly. Therefore you'll have to decide by yourself what the characteristics of outlying data points in your data set are. This could be for example: distance to the cluster center (threshold), local neighbourhood (a data point that has no/few data points in its neighgbourhood might be an outlier) or statistical characteristics (an outlier doesn't 'fit' to the distribution of a cluster, e.g. Grubbs' method). You can find a survey of outlier detection methods in A Survey of Outlier Detection Methodologies by Hodge and Austin (2004).

$\endgroup$
1
$\begingroup$

If you have a labelled training set of normal data (or mostly normal data), then it is easy to identify which clusters are "normal" and which are "anomalies": just associate each data point in the training set to a cluster. If a cluster appears many times in the training set, it is normal.

If you do not have a labelled training set, in the absence of additional information there is no way to declare some clusters as normal and some as anomalous.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.