2
$\begingroup$

For a school project, I'm planning to compare Spiking Neural Networks (SNNs) and Deep Learning recurrent neural networks, such as Long Short Term Memory (LSTMs) networks in learning a time-series. I would like to show some case where SNNs surpass LSTMs. Consequently, what are the limitations of LSTMs? Are they robust to noise? Do they require a lot of training data?

$\endgroup$
  • $\begingroup$ Any yes, LSTMs need an aweful lot of training data. See karpathy.github.io/2015/05/21/rnn-effectiveness for example. It was trained with Linux kernel code and got some results which look like C++, but they still fail to get some basic concepts (declare a variable before you use it; use a variable once declared). $\endgroup$ – Martin Thoma Mar 4 '16 at 18:25
  • $\begingroup$ Do you have good material for SNNs? As far as I know, they can't really be used for anything. It seems to me that SNNs are still in very early research. One point where SNNs are better than LSTMs is by modeling biological cells. $\endgroup$ – Martin Thoma Mar 4 '16 at 18:26
  • 1
    $\begingroup$ @MartinThoma this question is based off a project I'm doing for a course. If the results are good, I'll put them in a paper. If they're terrible, then I'll write a blog post. $\endgroup$ – Seanny123 Mar 4 '16 at 20:15
  • 1
    $\begingroup$ @MartinThoma I work in a lab that uses SNNs for biologically inspired AI. The first book on SNNs for scalable computation was published in 2003, so it's a pretty established area of research. For more info on my lab and how we're using SNNs, check out this video and ignore the fire alarm at the beginning. $\endgroup$ – Seanny123 Mar 4 '16 at 20:21
5
$\begingroup$

I finally finished the project. Given really short signals and a really small training set, SNNs (I used Echo State Machines and a neural form of SVM) vastly out-performed Deep Learning recurrent neural networks. However, this may be mostly because I'm really bad at training Deep Learning networks.

Specifically, SNNs performed better at classification of various signals I created. Given the following signals:

enter image description here

The various approaches had the following accuracy, where RC = Echo State Machine, FC-SVM = Frequency Component SVM and vRNN = Vanilla Deep Learning Recurrent Neural Network:

enter image description here

SNNs were also more robust to noise:

enter image description here

For more information, including how I desperately tried to improve the Deep Learning classification approach performance, check out my repository and the report I wrote which is where all the figures came from.

Update: After spending some time away from this project, I think one of the reasons that RNNs do horribly at this project is that they're bad at dealing with really long signals. Had I chunked the signals together with some sort of smoothing as preprocessing, they probably would have performed better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.