# Efficiently labelling training data in machine learning

Obviously it always depends on the specific case. However, my question is how to label data efficiently without writing a source code from scratch which solves the final problem itself?

For example, suppose that I want to use the $k$-Nearest Neighbours algorithm to classify tables from images according to their shape (square tables, rectangular tables, round tables).

For this purpose, I need to find let's say at least 500 images of various tables and then label them.

However, if I do not want to label them manually then I must write extensively a program to do this which essentially is the program I would like to have overall.

In other words, there is no point in writing from scratch an algorithm for accurately labeling data if I want to use machine learning finally because then this algorithm would solve my overall problem without machine learning. However, it is obvious that this algorithm would be very difficult to write to cover all the possible cases e.g. of tables from various images etc.

So how can I label data quickly without writing so many source code lines?

You can't. You can't squeeze blood out of a stone. There's no such thing as a free lunch. You can't get something from nothing. If you want labelled images, you will need to label them yourself, or find some existing data set that comes with labels. There are no shortcuts.

Yes, this is tedious and labor-intensive. This is one of the less-well-known and less-glamorous aspects of working on machine learning: in practical projects, we often spend the majority of our time (or more!) just assembling data sets, and only a small fraction on the actual learning algorithms themselves.

I know you could do various shortcuts, like trying to write a quick program to label them. But if that program makes mistakes, you'll just be training your machine learning algorithm to make the same mistakes -- so that's not actually helpful.

If you want to reduce the amount of labelling effort, there are ways to reduce the effort -- but still they will require a significant amount of manual labelling. Nothing comes for free. For instance, you can use active learning algorithms to identify which instances to label. A simple example of that is to manually label a few hundred images as your initial training set, train a classifier, apply it to all remaining images, pick out the 20 images that the classifier is least confident on, manually label those 20 images, add them to the training set, and repeat. (This is an application of uncertainty sampling.) There are other, more sophisticated methods out there as well.

Another plausible approach is to somehow cluster the images, then manually label a few images from each cluster. This has issues, though, as you'll need some reasonable way to cluster the images, and that might be a non-trivial task. (One possible approach for clustering is to take some existing, pre-trained ImageNet classifier -- e.g., VGG, Inception, ResNet, etc. -- throw away the last layer or last two layers, and use the output before those layers as the input to some clustering algorithm like k-means. Doing k-means clustering directly on the raw image probably won't work well, but if you do it on the vector of activation values at some deep layer near the end of a good pre-trained classifier, then you might get better results.)

Finally, we often use data augmentation to make the classifier more robust. For instance, for each image in the training set, we might generate 10 more copies, where each copy is rotated or translated or cropped by a random amount, and add that to the training set. This tends to help make the classifier more robust to changes in orientation or pose -- but it doesn't give you something for nothing. You still need a large training set that contains many different kinds of tables.

• Thank you very much for your answer! Finally, I have a very clear and thorough answer to my question. – Poete Maudit Jan 25 '18 at 9:09

There exist non machine-learning approaches to feature detection in images, but those are generally complicated and not very successful in comparison with machine learning methods.

Labeling data somehow without involving the algorithm to be trained is a hard requirement when doing supervised learning. As most decent automated labeling methods would use machine learning, this seems like a bootstrapping issue.

There also exists unsupervised learning methods, e.g. clustering approaches such as $k$-means. But most of these are unsuitable for very complex classification tasks such as image recognition. (Sure, Google did a reasonable large unsupervised image classification a few years a go, but they required computational resources that aren't available to most of us)

In the end, image recognition and other hard machine learning tasks are approached with ML methods precisely because humans are a lot better at them than computers, so we hope to 'learn' the computer to make decisions similar to the human. This means that, unfortunately, human effort is a required to train these algorithms.

• Thanks for your response. So what do you suggest to do for labelling data? Doing it manually and case by case or do you know any software that can facilitate this process? Otherwise, as I said, to write an extensive algorithm to label data is like trying to solve the main problem which you want to solve by training labeled data... – Poete Maudit Jan 24 '18 at 10:44
• @Universalis There might be some software available that makes the human classification job easier. That, however, is another question and not one suitable here. Perhaps you can ask that question on softwarerecs.stackexchange.com . – Discrete lizard Jan 24 '18 at 11:01
• Yes, I will ask this question there. But specifically how do you label data if not doing it manually? (I want to know for start how are people doing this) – Poete Maudit Jan 24 '18 at 11:14
• @Universalis I'm not sure what you're asking. You can cluster data and hope that the algorithm clusters objects that you would give the same label. You'd still have to name the label yourself, of course. But this approach isn't often successful in image classification. – Discrete lizard Jan 24 '18 at 11:22
• My question is very simple: how do you label data for training in machine learning to classify images if not doing it manually? – Poete Maudit Jan 24 '18 at 11:59

What you are looking for is very similar to a clustering task. So you should look into clustering algorithms for images (in the field of unsupervised learning), for example, the JULE algorithm (here and here) from 2016 does that sort of thing.

It is standard practice to label stuff by a human in the beginning. You could assist the human, but there is always a human component in creating a new dataset.

Interesting things to mention:

• AWS mechanical Turk
• Online learning
• Active learning