Take the 2-minute tour ×
Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It's 100% free, no registration required.

Having a unsupervised algorithm done on a dataset (e.g. K-means), how can I determine which cluster is normal and which is anomalous (for 30 clusters, for example)?

If the dataset contains normal traffic much more than anomalies (99%), then the radius of clusters associate to normal traffic is bigger. Is this true? Why?

If so, then we set the clusters with greater radius to normal according to a threshold. The smaller the threshold, the more the detection rate and the more the false alarm rate. Is this true?

Could a labeled dataset help determine the normal clusters? For example, finding the closest point to the centroid of a given cluster and checking whether it is labeled anomaly or not?

[1] uses Cluto software and its mountain visualization graph that I can't understand why its 3D, having feature space of dimension 41.

Reference: [1] Jose F. Nieves, Data Clustering for Anomaly Detection in Network Intrusion Detection

share|improve this question

migrated from security.stackexchange.com Nov 19 '12 at 13:25

This question came from our site for Information security professionals.

2 Answers 2

The radius depends on the number of clusers you choose. The more clusters, the smaller the radius.

If you have labelled data, you could eliminate the outliers/anomalies for determining the threshold radii for your clusters. For example you could take the greatest distance from the cluster center to its members (smaller radius for higher detection rate as stated in your question).

This question is closely related to the question Anomaly/outlier detection using fuzzy clustering. I am reposting my answer:

As far as I know there is no unique definition for what is an outlier/anomaly. Therefore you'll have to decide by yourself what the characteristics of outlying data points in your data set are. This could be for example: distance to the cluster center (threshold), local neighbourhood (a data point that has no/few data points in its neighgbourhood might be an outlier) or statistical characteristics (an outlier doesn't 'fit' to the distribution of a cluster, e.g. Grubbs' method). You can find a survey of outlier detection methods in A Survey of Outlier Detection Methodologies by Hodge and Austin (2004).

share|improve this answer

If you have a labelled training set of normal data (or mostly normal data), then it is easy to identify which clusters are "normal" and which are "anomalies": just associate each data point in the training set to a cluster. If a cluster appears many times in the training set, it is normal.

If you do not have a labelled training set, in the absence of additional information there is no way to declare some clusters as normal and some as anomalous.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.