String inputs in Machine Learning

Question

Several popular machine learning algorithms such as Logistic regression or Neural networks require its inputs to be numeric.

What I'm interested in is how you make these algorithms work on non-numeric inputs (such as short strings).

As an example, let's say we're building an email classification system (spam/not spam), where one of the input features is the sender address.

To be able to use a learning algorithm, we need to represent the sender address as a number. One way is to simply number the senders 1..n. Our training set might then look like this:

This won't work, however, because algorithms such as Logistic regression or Neural networks learn patterns in the input data, while in our example, the output looks totally random to the algorithm. Indeed, once in a university class, we tried to train a Neural network on a dataset that looked like this and the network was unable to learn anything (the learning curve was flat).

Would you use Logistic regression or Neural networks in this example at all? If yes, in which way? If not, what would be a good way to classify emails based on sender address?

A perfect answer would discuss the email classification example as well as handling short strings in ML in general.

Answer 1

One of the popular models is the Bag of Words model

Also, you can model the words as integers.. they have 'relative distance metrics' for that, and capture the very essence of the classification process. However a downside to that is the preprocessing step is expensive and also you need to have some domain knowledge.

A pretty famous distance metric is the Levenshtein distance which is based on the the number of single character edits. eg. that is, $d(walk,talk)<d(walk,plod)$.

The metrics depend upon the context of the classification process.. for example your distance metric for classifying rhyming words will be different from those designed to classify synonyms/words conveying similar meanings.For a list of string metrics, take a look at this wikipedia article.

Also, you can take a look at this review paper.