Just a note:

• rational-weighted recurrent $NN$s having boolean activation functions (simple thresholds) are equivalent to finite state automata (Minsky, "Computation: finite and infinite machines", 1967);

• rational-weighted recurrent $NN$s having linear sigmoid activation functions are equivalent to Turing Machines (Siegelmann and Sontag ???);

• real-weighted recurrent $NN$s having linear sigmoid activation functions are more powerful than Turing Machines ("Analog computation via neural networks", Siegelmann and Sontag, 1993);

but ...

• real-weighted recurrent $NN$s with Gaussian noise on the outputs cannot recognize arbitrary regular languages ("Analog Neural Nets with Gaussian or Other Common Noise Distributions Cannot Recognize Arbitrary Regular Languages", Maass and Sontag, 1995);

Just a note:

• rational-weighted recurrent $NN$s having boolean activation functions (simple thresholds) are equivalent to finite state automata (Minsky, "Computation: finite and infinite machines", 1967);

• rational-weighted recurrent $NN$s having linear sigmoid activation functions are equivalent to Turing Machines (Siegelmann and Sontag, "On the computational power of neural nets", 1995);

• real-weighted recurrent $NN$s having linear sigmoid activation functions are more powerful than Turing Machines (Siegelmann and Sontag, "Analog computation via neural networks", 1993);

but ...

• real-weighted recurrent $NN$s with Gaussian noise on the outputs cannot recognize arbitrary regular languages (Maass and Sontag, "Analog Neural Nets with Gaussian or Other Common NoiseDistributions Cannot Recognize Arbitrary Regular Languages"  , 1995);