# How to use frame based speech features for learning using a neural network classifier?

I am doing supervised learning on speech audio files using neural networks. For this purpose, I'll have to extract features from the audio file. But since an audio file is a time varying signal, it is generally divided into multiple frames and then features like MFCC etc are extracted from each frame. So how should I encode my feature vector for each training example(audio file) considering that it will be divided into different no of frames(depending on duration of file)?

• Sorry, I can't understand what you're asking. What research have you done? Do you understand how the MFCC features are typically computed? If you do, then what's the question? There's no "encoding" to be done; the MFCC features for each frame are already a 13-vector of real numbers, so if you have $n$ frames, you get $13n$ real numbers. Please edit your question to explain what research you've done, what you understand, what specifically you are confused about, and generally flesh it out so we can understand what you are confused about.
– D.W.
Apr 30 '15 at 6:50
• Thanks for your response D.W. I am a beginner in the field of ML and audio signal processing, so some of my understanding may be incorrect. What i meant to ask is that for each audio file I have 13n[i] features where n[i] is no of frames in the ith audio file. But to feed these to a neutral network, the number of features must be same for each audio file. But here they vary as per the no. of frames. So how should I take care of it?
– ksb
Apr 30 '15 at 12:00
• OK, that makes more sense. Please edit your question accordingly. Comments exist only to help you improve your question; all clarifications belong in the question, and shouldn't be left in only the comments. Also, I suggest you tell what what you want to use supervised learning for. In other words, what is the classification task? What property are you trying to learn/estimate? This will affect the best solution to this problem.
– D.W.
Apr 30 '15 at 19:55

Simple neural network as a structure doesn't have invariance across time scale deformation that's why it is impractical to apply it to recognize time series. To recognize time series usually a generic communication model is used (HMM). NN could be used together with HMM to classify individual frames of speech. In such HMM-ANN configuration audio is split on frames, frame slices are passed into ANN in order to calculate phoneme probabilities and then the whole probability sequence is analyzed for a best match using dynamic search with HMM.

To train HMM-ANN you need a segmentation of speech on states. HMM-ANN system usually requires initialization from more robust HMM-GMM system thus there are no standalone HMM-ANN implementation, usually they are part of a whole speech recognition toolkit. Among popular toolkits Kaldi has implementation for HMM-ANN and even for HMM-DNN (deep neural networks).

There are several more complex types of neural networks that are intended to model sequence data. They fall into class of recursive neural networks where connections have cycles. Recursive neural network can process sequences of features of arbitrary length. RNN systems are state of the art systems these days and you can train very accurate recognition system using them, however, training is not simple. You can check RNN toolkits like CURRENNT.

A case of recursive neural networks are long-short term memory networks. They are state of the art system for speech recognition these days. You can learn more about them from the following publication:

http://arxiv.org/pdf/1303.5778.pdf