I got a very hard automatically clustering problem with some training data (500 samples, each with roughly 5-50 classes and 10000-20000 data points in total). What I need to do is to cluster an input into multiple classes (the number of classes is unknown).

So far I had some several clustering algorithms, each gives me an output. I noticed that none of them works sufficient robust over the entire training set, but some works better than others on data with a small number classes, some works better on data with a large number of classes. Meanwhile, some gives me very good cluster results on part of a sample, while performs bad on the other half. Visually speaking, I feel that human should be able to generate a better clustering output by combining all initial outputs of using existing methods. However, it seems that it is hard to translate my human selection logic into codes. Because I know that a good cluster result in my data should be of a shape like a long eclipse, if I see a class in one of my output is very dislike this shape, I know there is something wrong, and if I look up the results of this area in other outputs, I can easily pick those close to my expectations. The actual case is even more complicated, my human decision on choose which class in which output for which region in my data sample uses both relative information among different outputs and prior knowledge on the distribution of tested data points. Please help me, I even donot know where to start.

Any comments and hints are welcome.

Thank you.

Thank you for your comments. I see what you said, but it turns out that my ultimate goal is different from what you suggested. I am not interested in finding which clustering output is better than others, but trying to generate a new output based on existing clustering output.

At the first glance, these two goals look compatible: if I can choose one best clustering output for a given set of data points, then I shall be able to repeat this process and apply it iterative to cover the entire dataset, and in this way we can generate a new output, whose solution is composed of different clustering outputs at different regions. However, in my problem this is really not a feasible approach, because I really donot know how to separate an entire dataset into reasonable subsets.

If we know how to do this job properly, the clustering problem can be largely simplified. We can first separate the entire set into nonoverlapping subset, then use an appropriate clustering method for each individual subset, and finally collect all subset clustering outputs as a new output. Meanwhile, it seems that this way of sequential process might be harmful if we make mistakes in the early stage of dividing subsets.

  • $\begingroup$ You say that the shape should be like a long eclipse. Do you know anything about the distribution of your data? The mention of a long eclipse makes me wonder if each cluster might be distributed according to a multivariate Gaussian distribution (as a scatterplot of samples drawn from such a distribution will look roughly like an eclipse). If so, have you tried using a Gaussian mixture model (e.g., using the EM algorithm to fit a Gaussian mixture model to the data)? That might produce a good clustering. $\endgroup$ – D.W. Dec 25 '13 at 19:22

If you can devise an algorithmic way to assess the quality of a clustering output, then one approach would be to run all algorithms on each data set, measure the quality of each, and use the one that is best.

So, you might want to focus on seeing if you codify your human intuition into a way to quantitatively measure the quality of a clustering output.

Are you familiar with a Gaussian mixture model? Your description of "long eclipse" makes me think that maybe that is what you are doing with your human intuition.

In particular, a scatterplot of samples drawn from such a a multivariate Gaussian distribution will look roughly like an eclipse. This makes me wonder if each cluster might perhaps be drawn from a multivariate Gaussian distribution (with different parameters for each cluster). If so, a Gaussian mixture model (e.g., using the EM algorithm to fit a Gaussian mixture model to the data) might produce a good clustering. Have you tried using that approach to clustering?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.