What restrictions apply to query and target vector encoding on fast-forward neural networks?

Question

I'm currently studying fast-forward multi-layer neural networks with back propagation, in the book I see that all query and target vectors are binary-encoded, this makes me believe that this is the only allowed encoding, yet all other neurons in the hidden layers and output layer can take on any real value between 0 and 1.

Now I am creating my own neural network and I want to classify an image by using the RGB values of every pixel as input neurons in some way.

So now my question is, as the title states, what restrictions apply to the encoding pattern used for query and target vectors?

Answer 1

The inputs are not restricted to be 0 or 1. They can be arbitrary real numbers. In practice we often normalize or standardize the inputs so that they are in the range $[-1,+1]$ (or most of the input values are in that range), but even that is not required, and even then, they can be any real number in that range.

The outputs are also real numbers, and they can be any real number: not limited to 0 or 1, and not limited to a particular range. (In some applications we interpret the outputs using the softmax function -- i.e., we postprocess the outputs of the neural network using another function -- which rescales them to get one likelihood value per output that's in the range $[0,1]$, but this is not mandatory and will depend on the specific application setting.)

Answer 2

Using highly variable data for training neural networks (NN) model may induce "overrating". So for practical reasons normalizing (or standardizing) the inputs in NNs can make training faster and reduce the chances of getting stuck in local optima. That the reason for normalizing data. A good choice for inputs is range [-1:1]. Another possibility is to scale the inputs to have mean 0 and a variance of 1.

Target normalization depends on the activation function used. For tanh is range [-1:1], for sigmoid function, for example, is [0:1], and the range arises from the nature of the function (output range). This is valid for continous data in input and output. An "advantage" of range [0:1] (sigmoid) is that output of ANN can be interpreted as a posterior probability.

Really, normalization for targets must be in the range [-0.9:0.9] or [0.1:0.9] (depending of the activation function used) to avoid the asympthotic parts of the function, regions where resolution is smaller.

For clasification problems as yours, output is discrete and not continous. You are two possibilites for your system:

1. example: 5 classes. Output is 1 unit with value continuous between 0 and 5 (normalized in the appropriate range as discussed above). Class is defined by the range where output is located (first class for target between 0 and 1, second class for target between 1 and 2, a so on, obviously in the normalized version);

2. example: 5 classes. 5 neurons target with discrete value 0 or 1. You train the net assigning 1 to correct class for each example and 0 for the other 4 neurons. Practice demonstrates that this option is better.

My suggestion: input normalized in the range [-1:1] (continuous variables given RGB values between 0 and 255, I think); output composed by n neurons (in the same number of your target categories for classifier), where only one unit target for each trainig example is assigned to 1 (all other to 0).

I hope I undestanded correctly your question.