# How is sound input and output data converted to use with machine learning networks?

Suppose one has a couple of .wav files with English spoken words, multiple ones for each word, and for each such set there exists a transcription of their right output, the pronunciation as ascii text.

As far as I know, machine learning neural networks use arrays of floats as input and output, and also internally.

What do one do in machine learning in order to convert such 'real world' data formats/data sets into another data structure that is meaningful and suitable for the machine learning neural networks?

Furthermore, what classifies a particular data structure as 'suitable', except the fact that it can be expressed as arrays of integers (what fits every digital data)?

(I suppose it could be more sophisticated than stripping the headers and feeding the uncompressed binary data in as integers, or is it?)

edit: in an other SE site's question (regarding how to filter out an image of Justin Bieber), an answer asserts that one "has some method of feature generation to transform the raw images into features that are useful for machine learning purposes", but it doesn't explain how this is done, or how does one begin to create a method for such a feature conversion.

Probably the most common way to represent audio for speech recognition is using the Mel-frequency cepstrum coefficients. If you're interested in finding out more about state of the art neural network based systems for speech recognition, I'd recommend checking out some of the recent work by George Dahl.

In regard to the other, more general portion of your question, using "binary" data in the way you describe is a very bad idea (if you mean what I think you mean). The binary representations used to store information in a computer is primarily designed for efficiency, both in terms of operations and space. Thus, this sort of representation is not necessarily representative of the actual characteristics of the data.

An example of this is the difference between signed and unsigned integers. If you're using 8 bit unsigned integers, then 11111111 > 00000001 but if you're using signed (2's complement) integers then 11111111 < 00000001. But, if you just represented the data as binary vectors, then you're essentially forcing a machine learning system to not only figure out whatever relationship you're trying to model, but also whatever encoding scheme you've used. You can avoid this by using representations that respect the characteristics (real valued, categorical, etc) of the data.

Lastly, if you're trying to predict 1 of $k$ mutually exclusive classes (such as characters), then the most common approach is to represent the class label (target) as a $k$ dimensional vector with a 1 at the $i$th position, corresponding to the class, and 0 everywhere else. You'll frequently see this referred to as a 1-of-$k$ or one-hot representation. You'll also want to use an appropriate output function for you're neural net, most likely the softmax function given by

$$f(x_i) = \frac{\exp(x_i)}{\sum_{j=1}^k \exp(x_j)},$$ where $x_i$, $i = 1, \ldots, k$, is the total input to the $i$th output unit. You can interpret this type of output as representing the conditional probability of each class given the input. The derivative is also particularly simple under the log-loss, which is what you should probably be using in this scenario.

In conclusion, neural networks are not magic black boxes. Just like any other ML technique, if you just throw data at them without considering the choices (inputs, outputs, loss function, etc) you're making and why, you'll probably have little success.

• This is a great answer. – András Salamon Jun 22 '13 at 15:19