I got a very hard automatically clustering problem with some training data (500 samples, each with roughly 5-50 classes and 10000-20000 data points in total). What I need to do is to cluster an input into multiple classes (the number of classes is unknown).
So far I had some several clustering algorithms, each gives me an output. I noticed that none of them works sufficient robust over the entire training set, but some works better than others on data with a small number classes, some works better on data with a large number of classes. Meanwhile, some gives me very good cluster results on part of a sample, while performs bad on the other half. Visually speaking, I feel that human should be able to generate a better clustering output by combining all initial outputs of using existing methods. However, it seems that it is hard to translate my human selection logic into codes. Because I know that a good cluster result in my data should be of a shape like a long eclipse, if I see a class in one of my output is very dislike this shape, I know there is something wrong, and if I look up the results of this area in other outputs, I can easily pick those close to my expectations. The actual case is even more complicated, my human decision on choose which class in which output for which region in my data sample uses both relative information among different outputs and prior knowledge on the distribution of tested data points. Please help me, I even donot know where to start.
Any comments and hints are welcome.
Thank you for your comments. I see what you said, but it turns out that my ultimate goal is different from what you suggested. I am not interested in finding which clustering output is better than others, but trying to generate a new output based on existing clustering output.
At the first glance, these two goals look compatible: if I can choose one best clustering output for a given set of data points, then I shall be able to repeat this process and apply it iterative to cover the entire dataset, and in this way we can generate a new output, whose solution is composed of different clustering outputs at different regions. However, in my problem this is really not a feasible approach, because I really donot know how to separate an entire dataset into reasonable subsets.
If we know how to do this job properly, the clustering problem can be largely simplified. We can first separate the entire set into nonoverlapping subset, then use an appropriate clustering method for each individual subset, and finally collect all subset clustering outputs as a new output. Meanwhile, it seems that this way of sequential process might be harmful if we make mistakes in the early stage of dividing subsets.