# Non linear neural networks?

The activation of a perceptron style neuron is:

$DotProduct(Inputs, Weights)+Bias > 0$

That is essentially classifying what side of a (hyper)plane a point is on (positive or negative side), like the below:

$DotProduct(Point, Normal)+D > 0$

Looking at an N input, M layer perceptron network that has a single output can be seen as classifying a point as inside or outside some hyper shape.

But, it seems like all the tests are linear, and switching to sigmoid activation doesn't seem to add a whole lot of non linearity - like if you were visualizing this NN shape, even a sigmoid activated network would be mostly made up of planes, not curved surfaces.

On the other hand we have support vector machines as a machine learning tool, and using things like the "kernel trick" which can make a non linear classification of points for being inside or outside.

This makes me wonder, is there a version of neural networks that has more free form separation shape?

It seems like it would be possible to have a better performing neural network with fewer neurons and/or layers.