How would a neural network deal with an arbitrary length output?

Question

I've been looking into Recurrent Neural Networks, but I don't understand what the architecture of a neural network would look like when the output length is not necessarily fixed.

It seems like most networks I've read descriptions of require the output length to be equal to the input length or at least a fixed size. But how would you do something like convert a word to the string of corresponding phonemes?

The string of phonemes might be longer or shorter than the original word. I know you could sequence in the input characters using 8 input nodes (bitcode of the character) in a recurrent network, provided there's a loop in the network, but is this a common pattern for the output stream as well? Can you let the network provide provide something like a 'stop codon'?

I suppose a lot of practical networks, like those that do speech synthesis should have an output that is not fixed in length. How do people deal with that?

Answer 1

You just add more neurons. You're right that it's often not useful (if you only get 5 bits of information in, it's hard to put 10 bits of information out), but if you want to (e.g. because your output format is less dense), go ahead.

As a trivial example, if you wanted to create an ANN to convert characters to graphemes (as represented on an 8x8 grid of on/off pixels) you could have 26 input neurons and 64 output neurons. Connect each input neuron to the output neurons which would be turned on if that character is being displayed; then the output neurons' function is just logical OR.

The standard learning algorithms like gradient descent should work fine no matter the size of the output layer.

---

EDIT: I guess your question is: "how do you handle unknown input and output lengths?" Any Turing machine can be simulated by a recurrent neural network, so you will never run into a "halting problem" (if I understand what you mean by that phrase).

It's very rare that you can't bound the size of the output as a function of the input size. So you just have some metalearning procedure which generates the network for you on the fly.

One common idea is that of "template" models. I'm not familiar with text-to-speech, but I guess you would make an assumption like "no phoneme spans more than 6 characters". So you build your network with 6 inputs, and just repeat it for as long as the word is. You need some rule like "each network gives the pronunciation of the middle two characters" to handle overlaps, as well as some special handling for the beginning and end of words too.