Standardizing Data for Neural Networks

Question

Let's say we have a data set with following features [age, sex, country, city, annual income] [35, male, USA, New York, 73000].

I came across the article which explains how to deal with strings as an NN input.

Article example showing how to standardize data:

[0] 30  male    38000.00  urban     democrat
[1] 36  female  42000.00  suburban  republican
[2] 52  male    40000.00  rural     independent
[3] 42  female  44000.00  suburban  other

[0] -1.23  -1.0  -1.34  ( 0.0   1.0)  (0.0  0.0  0.0  1.0)
[1] -0.49   1.0   0.45  ( 1.0   0.0)  (0.0  0.0  1.0  0.0)
[2]  1.48  -1.0  -0.45  (-1.0  -1.0)  (0.0  1.0  0.0  0.0)
[3]  0.25   1.0   1.34  ( 1.0   0.0)  (1.0  0.0  0.0  0.0)

Now let's try to use this approach in our example where we could end up with thousands of different cities. We would have to create at least N vectors size of [1 ... N]. That approach just does not seem to be very efficient when we have unlimited number of possibilities.

That was a brief explanation of standardizing data problem. Now let me expand it to my specific NN task. I have a data set with 30+ features. Features consist of real numbers, decimal numbers, strings. All of them are relevant.

I have to transform non-computable data into numeric, NN computable form. The question is how to standardize a set of data with many features of different types (strings and numbers)?

Answer 1

You can distinguish between the following types of data (source, see Level of measurement for something similar):

• Nominal: You can't calculate with them. They only support checks for identity, e.g. names.
• Ordinal: You can order them, but not compare the distance. Examples are grades or sizes for cloths.
• Interval scaled: Additionally to identity checks and ordering it supports addition. Examples are temperature in °C / °F
• Ratio scaled: They support multiplication in addition to the ones before. Examples are mass and length.

For neural networks, ratio scaled feature make most sense. You can sometimes change your view on things to get it, but most of the time you cannot simply transform an ordinal scaled feature to be ratio scaled.

Two tricks for your example come to my mind:

• 1-hot encoding: Instead of using one number for the country (e.g. 1 for Australia, 2 for Austria, 3 for France, 4 for Germany, ...) you should use a 1-hot encoding (e.g. one bit per country, the bit is only set if it is this country). I am pretty sure this will work much better, although your feature vector will get much bigger. (Attention: Depending on how much data you have, you might suffer from overfitting. You could also partition your dataset according to this feature. So, for example, you might have a lot of data for the US, for France and Germany. But you only have a little data for african countries. Then you could simply have one bit for each of US, France, Germany and complete Africa)
• Latitude / Longitude: Instead of using the citys directly, you can use the citys coordinate. This makes your feature even more meaningful and you have two floats for it which are ratio scaled.

You might also think about transforming the country to something more meaningful for your problem (e.g. membership in an organization like OECD).

You could also combine different classifiers. So you could use the "bad scaled features" for a decision tree and feed the result into a NN.