Disadvantages to using simple step functions for activation in neural networks?

Question

From what I have read, the main advantage to using tanh(x) or sigmoid(x) as an activation function for neural networks is that it is very easily differentiable.

I am trying to implement a neural network that uses a genetic algorithm for optimisation rather than backpropogation, and therefore it doesnt matter if my activation function is differentiable or not

When running my algorithm on matlab/octave, it seems like there is a massive bottleneck in the computation when it is trying to calculate sigmoid of the input to the neuron, so I was wondering if there would be any disadvantage to using a simple step function instead of the more complex sigmoid or tanh functions to activate the neuron?

Answer 1

The disadvantage would be that your fitness landscape would look less smooth (a small change in your weights may result in a big change in the output or no change at all). This may affect the convergence speed of your GA, since small changes in the chromossomes may not produce differences in fitness, rendering it blind to some benefical changes. If your problem is performance, I'd suggest to try the hard ReLU activation function instead of a sigmoid one.