2
$\begingroup$

The activation of a perceptron style neuron is:

$DotProduct(Inputs, Weights)+Bias > 0$

That is essentially classifying what side of a (hyper)plane a point is on (positive or negative side), like the below:

$DotProduct(Point, Normal)+D > 0$

Looking at an N input, M layer perceptron network that has a single output can be seen as classifying a point as inside or outside some hyper shape.

But, it seems like all the tests are linear, and switching to sigmoid activation doesn't seem to add a whole lot of non linearity - like if you were visualizing this NN shape, even a sigmoid activated network would be mostly made up of planes, not curved surfaces.

On the other hand we have support vector machines as a machine learning tool, and using things like the "kernel trick" which can make a non linear classification of points for being inside or outside.

This makes me wonder, is there a version of neural networks that has more free form separation shape?

It seems like it would be possible to have a better performing neural network with fewer neurons and/or layers.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Sigmoid does add nonlinearity -- and the nonlinearity "builds" as you add more layers. Neural networks can recognize arbitrary decision boundaries (given enough neurons and enough layers).

I think your visualization of what the sigmoid activation does is a bit off. Take a look at http://playground.tensorflow.org/ and play with some of the tuning knobs to see the kinds of decision boundaries that can be achieved -- that site is a helpful way to get some intuition.

So, the answer to your question is: we don't need a separate version of neural networks. Standard neural networks already can support arbitrary decision boundary shapes.

[ About the kernel trick: The kernel trick can be used here, but is possibly a bit more expensive than in the case of SVM's; for SVM's, there's a particular mathematical trick that avoids the need to generate all of the "extra features", but that trick is specific to SVM's. You can still construct derived features and add them to the feature vector before applying a neural network, though, if you want -- nothing prevents you from doing it. But the kernel trick isn't needed, because neural networks can learn non-linear functions. In contrast, without the kernel trick, SVMs can only learn linear decision boundaries, so it's needed more there. ]

Improve this answer
$\endgroup$
2
  • $\begingroup$ D.W. This seems very related but let me know if you prefer I ask a new question. While a non linear activation function and multiple layers can do non linear classification, do you know of any NN setups that allow for more direct non linear classification? Also, I'm wondering, are there any NN setups or research where you can multiply neuron inputs instead of sum them? That seems like it would be helpful for non linearity. For instance, having 2 weights per neuron output. One for addition, one for multiplication perhaps? $\endgroup$
    – Alan Wolfe
    Commented Feb 5, 2017 at 16:11
  • $\begingroup$ Actually, just found "Sum-Product Networks: A New Deep Architecture". Very cool! turing.cs.washington.edu/papers/uai11-poon.pdf $\endgroup$
    – Alan Wolfe
    Commented Feb 5, 2017 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.