From what I have read, the main advantage to using tanh(x) or sigmoid(x) as an activation function for neural networks is that it is very easily differentiable.
I am trying to implement a neural network that uses a genetic algorithm for optimisation rather than backpropogation, and therefore it doesnt matter if my activation function is differentiable or not
When running my algorithm on matlab/octave, it seems like there is a massive bottleneck in the computation when it is trying to calculate sigmoid of the input to the neuron, so I was wondering if there would be any disadvantage to using a simple step function instead of the more complex sigmoid or tanh functions to activate the neuron?