I currently studying machine learning and pattern recognition area. Today, my professor said implementing an unsupervised system that automatically labels data is difficult. Why is that?

I think if I am given a data set, then I can categorize all data into groups. Then, for unknown input data, I can extract features and put it into the group that best fits all the features. Can anyone explain whats wrong with my intuition and why is it difficult problem?

  • 1
    $\begingroup$ When you are dealing with low-dimensional data (scalars) you may be able to classify by eye, but as soon as you start dealing with highly dimensional data [which nearly all the sciences deal with, astronomic data, genomic data, high energy physics data] it gets very difficult very quickly. $\endgroup$ Jun 12, 2014 at 8:24

2 Answers 2


The problem with your intuition is that, frankly, you don't have one. At least not a useful, that is algorithmic one, one that tells you what is hard for computers and what is not. You think in terms of what you can do with every-day data -- but that's not an appropriate frame of reference for this problem. As evidence, consider this question.

Some specific concerns:

  1. "I can" != "I can build an algorithm which can".
  2. Can you? Look at some data dumps from CERN and try to classify them.
  3. What exactly are "features" (mathematically)?
  4. What does "fits best" mean (mathematically)?

I recommend three things that will help you build a (better) intuition.

  1. Study computability and complexity theory -- this will help you build intution for hardness of computational problems.
  2. Program things -- this will help you build intution for hardness of implementing things.
  3. Try to build an unsupervised learner and fall into all the pitfalls.

implementing any nontrivial machine learning problem is difficult! but there are some basic different levels of difficulty (a continuum/hierarchy so to speak). the professor is contrasting the problem with another problem. consider these two problems. image classification is a common challenging problem at the edge of ML feasibility but of course not the only kind of classification problem; the example is based on it.

  • given a set of images, find out distinct objects that are in the images. ie the result is a set of arbitrary choices about which category each images is in. no information is given about categories at all, not even how many categories there are.
  • given a set of images and finite set of labels ("categories"), classify the images based on the labels. one has training data such that the images are correctly classified. classify the images in the test set by choosing one of the finite known labels.

clearly in option B there is more data to work with, it is less arbitrary, so the ML algorithm can potentially be more successful, and also there are infinite possible labels in option A.

this basic difference in challenge is why the recent Google results that found image classifications without labels in the training data is considered such a dramatic milestone/ breakthrough in ML where even the tabloid headlines were not necessarily overhyped!

  • $\begingroup$ (for some rough intuition/analogy imagine option A like a freeform essay test & option B like a multiple choice quiz, with example/sample answers provided! in option B the "extra data to work with" is the labels & known training categorizations.) $\endgroup$
    – vzn
    Jun 12, 2014 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.