Just a note:
rational-weighted recurrent $NN$s having boolean activation functions (simple thresholds) are equivalent to finite state automata (Minsky, "Computation: finite and infinite machines", 1967);
rational-weighted recurrent $NN$s having linear sigmoid activation functions are equivalent to Turing Machines (Siegelmann and Sontag ???);
real-weighted recurrent $NN$s having linear sigmoid activation functions are more powerful than Turing Machines ("Analog computation via neural networks", Siegelmann and Sontag, 1993);
but ...
- real-weighted recurrent $NN$s with Gaussian noise on the outputs cannot recognize arbitrary regular languages ("Analog Neural Nets with Gaussian or Other Common Noise Distributions Cannot Recognize Arbitrary Regular Languages", Maass and Sontag, 1995);