Deep Learning, now one of the most popular fields in Artificial Neural Network, has shown great promise in terms of its accuracies on data sets. How does it compare to Spiking Neural Network. Recently Qualcomm unveils its zeroth processor on SNN, so I was thinking if there are any difference if deep learning is used instead.
Strictly speaking, "Deep" and "Spiking" refer to two different aspects of a neural network: "Spiking" refers to the activation of individual neurons, while "Deep" refers to the overall network architecture. Thus in principle there is nothing contradictory about a spiking, deep neural network (in fact, the brain is arguably such a system).
However, in practice the current approaches to DL and SNN don't work well together. Specifically, Deep Learning as currently practiced typically relies on a differentiable activation function and thus doesn't handle discrete spike trains well.
Real neurons communicate via discrete spikes of voltage. When building hardware, spiking has some advantages in power consumption, and you can route spikes like data packets (Address Event Representation or AER) to emulate the connectivity found in the brain. However, spiking is a noisy process; generally a single spike doesn't mean much, so it is common in software to abstract away the spiking details and model a single scalar spike rate. This simplifies a lot of things , especially if your goal is machine learning and not biological modeling.
The key idea of Deep Learning is to have multiple layers of neurons, with each layer learning increasingly-complex features based on the previous layer. For example, in a vision setting, the lowest level learns simple patterns like lines and edges, the next layer may learn compositions of the lines and edges (corners and curves), the next layer may learn simple shapes, and so on up the hierarchy. Upper levels then learn complex categories (people, cats, cars) or even specific instances (your boss, your cat, the batmobile). One advantage of this is that the lowest-level features are generic enough to apply to lots of situations while the upper levels can get very specific.
The canonical way to train spiking networks is some form of Spike Timing Dependent Plasticity (STDP), which locally reinforces connections based on correlated activity. The canonical way to train a Deep Neural Network is some form of gradient descent back-propagation, which adjusts all weights based on the global behavior of the network. Gradient descent has problems with non-differentiable activation functions (like discrete stochastic spikes).
If you don't care about learning, it should be easier to combine the approaches. One could presumably take a pre-trained deep network and implement just the feed-forward part (no further learning) as a spiking neural net (perhaps to put it on a chip). The resulting chip would not learn from new data but should implement whatever function the original network had been trained to do.
The distinction between SNNs and Deep Learning, especially in Spaun, is a gray area. Spaun uses modified Deep Learning techniques in it's vision system for digit recognition. Essentially, it uses a spiking version of a Constitutional Neural Network.
More recently, the same technique of converting Deep Learning approaches to spiking neurons for use in neuromorphic hardware (such as Spinnaker and Brainstorm for lower latency and greater power efficiency) have been applied to Convolutional Neural Networks.
The differences go a lot deeper than what I've mentioned here (like how SNNs can approximate dynamic systems and other non-linear functions without training), but unfortunately there's no easy way to summarize them. If you want more details, check out "How to Build a Brain" by Chris Eliasmith for a greater overview of how SNNs are being used to create Artificial Intelligence.