3
$\begingroup$

Why is every artificial neural network layered? Why isn't each node just a separate process?

$\endgroup$
  • $\begingroup$ If you mean process in the sense of operating systems, that would be an extremely inefficient way of implementing a neural network. The running time would be overwhelmingly dominated by interprocess communication overheads. $\endgroup$ – David Richerby Sep 21 '16 at 20:44
4
$\begingroup$

A neural net will work with any arbitrary topology, as long as it has no cycles. We often use layers because it is easy to implement and because it allows a human to set parameters for how large the net should be, such as "four layers with 5, 6, 7 and 10 nodes, respectively".

I cannot find anything on the web about neural networks that are not organised into layers, but there is nothing principally impossible about them. Here's an idea for a rainy sunday afternoon: implement a neural net as a directed acyclic graph and use an evolutionary algorithm to find a good topology. Compare it to a layered neural net with similar numbers of nodes and edges.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

There are some hand wavy intuitions that adding depth (stacked layers) allows the formation and combination of higher level features. So you can take features from the layer before and build more complex ones by combining them together, rather than starting from scratch.

Conv nets are the poster child for this. You can see that their low level layers learn things like gabor filters and the deeper layer learn things like face detectors.

https://arxiv.org/abs/1512.03965 https://arxiv.org/abs/1602.04485

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ But why layers though? Why not some other topology in which nodes in "layer" one talk to nodes in layers 2, 3, and 4, and nodes in "layer" 2 talking to nodes much further ahead, too? I conjecture that the slowdown does not compensate for any possible gains in classification performance, but honestly I don't know. $\endgroup$ – Lieuwe Vinkhuijzen Sep 21 '16 at 20:20
  • $\begingroup$ oh... i see. I misinterpreted the question. A reason may be how matrix multiplication works. A matmul is a linear combination of basis vectors, aka a linear combination of lower level features. Also, matrix multiplications are fast $\endgroup$ – Alexander Telfar Sep 22 '16 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.