7
$\begingroup$

Deep Learning, now one of the most popular fields in Artificial Neural Network, has shown great promise in terms of its accuracies on data sets. How does it compare to Spiking Neural Network. Recently Qualcomm unveils its zeroth processor on SNN, so I was thinking if there are any difference if deep learning is used instead.

$\endgroup$
  • 3
    $\begingroup$ I am not an expert in AI, but this question has the feel of: "This apple pie is quite good. What would be the key differences if we used pears instead?" $\endgroup$ – Raphael Feb 10 '14 at 8:42
  • 2
    $\begingroup$ Before asking questions here, you are expected to do some research and try to answer the question yourself. What did you look at? Is there some specific aspect of SNNs that you want to compare to deep learning or are you baking pies with @Raphael? :-) $\endgroup$ – David Richerby Feb 10 '14 at 8:45
  • $\begingroup$ the main pt with deep learning is more the length of time & (large) # of neurons. one can use different neural net architectures. have not heard of SNN used for deep learning yet but in principle it seems it could be tested by someone. it is not really known yet why some neural algorithms seem to scale better for deep learning than others. "In practice, there is a major difference between the theoretical power of spiking neural networks and what has been demonstrated. They have proved useful in neuroscience, but not (yet) in engineering." $\endgroup$ – vzn Feb 10 '14 at 16:26
  • $\begingroup$ @vzn SNNs built using the Neural Engineering Framework have been applied in diverse engineering applications such as arm and quadrocopter control. $\endgroup$ – Seanny123 Mar 4 '16 at 20:28
11
$\begingroup$

Short answer:

Strictly speaking, "Deep" and "Spiking" refer to two different aspects of a neural network: "Spiking" refers to the activation of individual neurons, while "Deep" refers to the overall network architecture. Thus in principle there is nothing contradictory about a spiking, deep neural network (in fact, the brain is arguably such a system).

However, in practice the current approaches to DL and SNN don't work well together. Specifically, Deep Learning as currently practiced typically relies on a differentiable activation function and thus doesn't handle discrete spike trains well.

Further details:

Real neurons communicate via discrete spikes of voltage. When building hardware, spiking has some advantages in power consumption, and you can route spikes like data packets (Address Event Representation or AER) to emulate the connectivity found in the brain. However, spiking is a noisy process; generally a single spike doesn't mean much, so it is common in software to abstract away the spiking details and model a single scalar spike rate. This simplifies a lot of things , especially if your goal is machine learning and not biological modeling.

The key idea of Deep Learning is to have multiple layers of neurons, with each layer learning increasingly-complex features based on the previous layer. For example, in a vision setting, the lowest level learns simple patterns like lines and edges, the next layer may learn compositions of the lines and edges (corners and curves), the next layer may learn simple shapes, and so on up the hierarchy. Upper levels then learn complex categories (people, cats, cars) or even specific instances (your boss, your cat, the batmobile). One advantage of this is that the lowest-level features are generic enough to apply to lots of situations while the upper levels can get very specific.

The canonical way to train spiking networks is some form of Spike Timing Dependent Plasticity (STDP), which locally reinforces connections based on correlated activity. The canonical way to train a Deep Neural Network is some form of gradient descent back-propagation, which adjusts all weights based on the global behavior of the network. Gradient descent has problems with non-differentiable activation functions (like discrete stochastic spikes).

If you don't care about learning, it should be easier to combine the approaches. One could presumably take a pre-trained deep network and implement just the feed-forward part (no further learning) as a spiking neural net (perhaps to put it on a chip). The resulting chip would not learn from new data but should implement whatever function the original network had been trained to do.

$\endgroup$
  • 1
    $\begingroup$ I recently saw a presentation on a SNN that was trained with only a few dozen images and was amazingly accurate given the small sample size. Given advantages like that, are people working on ways to do something like deep learning with SNNs? Would the approach described in this recent paper arxiv.org/abs/1407.7906 "How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation" make it realistic to do Deep Learning with SNNs (since it doesn't rely on back propagation)? $\endgroup$ – Jason Livesay Aug 16 '14 at 10:05
5
$\begingroup$

The largest application of SNNs that I know of is Spaun, whose neural networks were built using the Neural Engineering Framework and the Nengo neural simulator.

The distinction between SNNs and Deep Learning, especially in Spaun, is a gray area. Spaun uses modified Deep Learning techniques in it's vision system for digit recognition. Essentially, it uses a spiking version of a Constitutional Neural Network.

More recently, the same technique of converting Deep Learning approaches to spiking neurons for use in neuromorphic hardware (such as Spinnaker and Brainstorm for lower latency and greater power efficiency) have been applied to Convolutional Neural Networks.

The differences go a lot deeper than what I've mentioned here (like how SNNs can approximate dynamic systems and other non-linear functions without training), but unfortunately there's no easy way to summarize them. If you want more details, check out "How to Build a Brain" by Chris Eliasmith for a greater overview of how SNNs are being used to create Artificial Intelligence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.