I would like to know whether it is computationally possible for a computer to go through the words of an audiobook as input, output a file containing both the original audio and the text corresponding to each word (which could be reviewed by a human), and a silent software tag between each word marking each word boundary.

How much of this process could be automated, and how much of it would have to be carried out by a human?

This process would yield a file where instead of just being able to play, pause, and move forward and backwards by a given number of minutes, as could be done in the original audiofile, we could now also provide the following functionality:

  • move back and forth a given number of words
  • move back and forth a given number of sentences
  • move back and forth a given number of chapters (or subsections)
  • spell the current word
  • look up the current word in a dictionary
  • accurately repeat over and over, the text between any two markers, with a custom number of seconds between each repeat (which is good for learning / memorizing stuff)

Note that while audio books were primarily designed for blind people, has increased functionality would both give them more power as well as empower people with learning disabilities or both.

UPDATE: Since the audiobook is created from a text (either a text file or a textbook which can be read using OCR), I wish the case to be considered where the source words and punctuation are available. My problem is having the computer assign a time t_(w_i) to the beginning of each word w_i and store these in the output sound+wordTimes file (which could be post-edited for inaccuracies by humans). I have not seen anything that does this so far.

This link seems to suggest that what I am trying to do is possible (via a technique called forced alignment).

  • $\begingroup$ There is a good selection of software to do that, you can find some links here: stackoverflow.com/questions/34983925/… $\endgroup$ – Nikolay Shmyrev Feb 7 '16 at 9:11
  • $\begingroup$ Thanks, Nicolay, for your link. My only concern is that your link points to software libraries that do this, but I am looking for a full-fledged Android app. $\endgroup$ – Jack Maddington Feb 7 '16 at 16:27
  • 1
    $\begingroup$ I'm afraid that requests for software packages, libraries, or toolkits are off-topic here. Questions about algorithms, techniques, or methods are potentially suitable here, but it's important to tell us what research you've done, what you've tried, and what approaches you've considered and rejected. The obvious approach is to apply speech recognition tools (which can tell you what words they recognized and when each word occurred), then match those up against the known sequence of source words, and fix up any errors. $\endgroup$ – D.W. Feb 7 '16 at 17:57
  • 1
    $\begingroup$ Also, thank you for editing the question, but in the future please don't write "Stuff... UPDATE: More stuff". Instead, edit the question to be what it should have been from the start, so it reads coherently and logically from start to finish. No need to mark what has changed -- we have revision control for that. $\endgroup$ – D.W. Feb 7 '16 at 17:58

The fact that humans can understand audio books shows that it is computationally possible. This does not make the problem easy for an AI. It also does not mean that you can write a formalism of the language in audio form for parsing. Anything involving audio / recordings is going to involve lots of fuzzy data. This is not generally what someone would call parsing, but instead speech recognition. If you can properly recognize words in the correct context then the rest of the stuff you mentioned like dictionary lookup, and moving around a book is basically trivial.

It is easier to do this if you already have the words though. I would recommend something like splitting the audio based on breaks, and then applying speech recognition on each section to make sure the word matches the audio that was split. This is to check if errors in splitting exist. If there is an error where the word does not match the audio a human could be prompted to manually split them.

  • $\begingroup$ You've considered the case where the source text is not available (hence you mentiin speech recognition alone). But what if the words were available. Would assigning a time to each word be trivial? (please see my updated post). $\endgroup$ – Jack Maddington Feb 6 '16 at 18:23
  • $\begingroup$ Nothing is trivial when it comes to AI recognition tasks like this. If you had an audio file exactly annotated with the words, and their exact times, it would be trivial. The source text will help assign, but that's till not going to be trivial by a long shot. $\endgroup$ – jmite Feb 7 '16 at 6:42
  • $\begingroup$ The word trivial was in reference to UI things like moving around a book. $\endgroup$ – 44701 Feb 7 '16 at 6:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.