I have made and trained a simple neural network which now seems to produce outputs reasonable in all the aspects but one: it gives negative values from time to time even though the outputs are always either 0 or 1 (positive) in all my treaining samples (the actual neural outputs show to be more or less close to the answer most probable but never actually reach them, I consider this ok). Won't "telling" the network what output values are just impossible improve its performance.
Perhaps my mistake is in choosing activation functions. As far as my input values domain is (-1, 1) I use TANH for input neurons, but I have no idea what is the logic to choose hidden and output neurons activation functions and so I set the same (TANH) for all of them (update: I have just noticed that I have mistakenly chosen linear activation function for the output layer while inpit and hidden layers use TANH).
Clearly a negative value means "the answer is very probable to be zero" in this case and it works just fine from the practical point of view but it still gives me a sense that I am doing something wrong.