The activation of a perceptron style neuron is:
$DotProduct(Inputs, Weights)+Bias > 0$
That is essentially classifying what side of a (hyper)plane a point is on (positive or negative side), like the below:
$DotProduct(Point, Normal)+D > 0$
Looking at an N input, M layer perceptron network that has a single output can be seen as classifying a point as inside or outside some hyper shape.
But, it seems like all the tests are linear, and switching to sigmoid activation doesn't seem to add a whole lot of non linearity - like if you were visualizing this NN shape, even a sigmoid activated network would be mostly made up of planes, not curved surfaces.
On the other hand we have support vector machines as a machine learning tool, and using things like the "kernel trick" which can make a non linear classification of points for being inside or outside.
This makes me wonder, is there a version of neural networks that has more free form separation shape?
It seems like it would be possible to have a better performing neural network with fewer neurons and/or layers.