3
$\begingroup$

Does anyone know if Deep Learning Bots can already, for example, train on many books of an author and output a similar but new book?

I've been wanting to get into ML for quite a while but was lacking a project to use as an incentive to keep learning, and this specific AI has caught my interest. It seems feasible from the little I know of ML so far (I'm a beginner who has taken the initial module of Deeplearining.ai course from Coursera), but so far from what I can scoop out of the internet it doesn't seem like AIs are quite there yet. They seem to be somewhat convincing, but sometimes a weird outlier appears in the text; strangely, music composition seems more convincing. Does anyone disagree?

Another relevant question: is this too difficult for a beginner? I've graduated from engineering 1 year ago, so I have some ease with programming, but I don't know how difficult Recurrent Neural Networks (RNN) and Natural Language Programming (NLP) can be.

Also, on a sidenote, does a more experienced programmer have a suggestion of path I should take to learn the necessary skills to program such a bot, i.e., online courses and books?

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ This is far beyond the capabilities of current methods. $\endgroup$ – Yuval Filmus Jul 27 '18 at 19:27
  • $\begingroup$ In theory a word-level RNN would be able to do something like this. But it would take a ridiculous amount of training data (much more than a single author's books) and a ridiculous amount of LSTM (much more than a desktop computer can reasonably simulate). Practically, it's impossible at present, even if you had all of Google and Microsoft's resources to devote to it. $\endgroup$ – Draconis Jul 27 '18 at 19:48
5
$\begingroup$

Researchers are trying it out, but RNNs learn character-by-character (sequences of characters), so it is difficult to get something that resembles a story plot, as a whole. This link by a Stanford researcher explains the current (2015) state of the art. Here is the code and here are some data sets for you to start experimenting with.

Another experiment involving the writing of a travel blog also concluded that you cannot really create a long passage that makes sense at the moment. The recommendation was to look at the word level, rather than character, and focus on something more manageable, such as sentence autocompletion.

When it comes to words, the more unique ones there are in the source data set (for example using the Game of Thrones books), the tougher it becomes to train a good model. Suggestions are to limit input to more basic words (think children's vocabulary) and to have at least a total training sample at least 100 times larger than the desired output.

A trained neural network could perhaps output smaller texts (<10000 words) that make some sense, if they are of a rather structured nature. A whole book needs both a coherent plot from start to finish, as well as twists in-between. As a result, it is still way too difficult.

This guy tried it out and discovered that the AI would get sometimes stuck in loops. Furthermore, the draft produced did not always make sense in terms of the storyline and required heavy human editing.

Here is another example of an LSTM RNN outputting a Harry Potter chapter, where the sentences are grammatically correct, but sometimes make no sense.

If you want to learn more about RNNs, the Deep Learning book by Goodfellow, Bengio, and Courville comes highly recommended and has a relevant chapter.

For something more specific to writing, The Bestseller Code book uses text mining techniques and should be an interesting read.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ I nominate this answer for "best turnaround". Filip, I hope you won't get too discouraged by the downvotes on your initial answer; after the revisions, this is much better and a very helpful answer! Thank you for putting the work into improving your answer. I hope you'll continue to contribute here. (Cc: @Evil) $\endgroup$ – D.W.♦ Aug 7 '18 at 0:33
  • $\begingroup$ @DavidRicherby, looks like the answer has been edited to drastically improve it; I thought you might like to know. $\endgroup$ – D.W.♦ Aug 7 '18 at 0:33
  • 1
    $\begingroup$ Wow, it is indeed the best turnaround, +1 from me. $\endgroup$ – Evil Aug 7 '18 at 2:22
  • 1
    $\begingroup$ Thank you for the answer! I happened to also have found most of the informationduring this first week of my deep dive into machine learning, but that's a much more comprehensive and organized answer than the jumbled mess in my beginner's mind. $\endgroup$ – Philippe Fanaro Aug 7 '18 at 3:15
  • $\begingroup$ @D.W. Thanks for the heads up. -1 now +1. $\endgroup$ – David Richerby Aug 7 '18 at 13:08
3
$\begingroup$

No. As Yuval writes, this is way beyond what we know how to achieve with the state of the methods known today. Definitely too difficult for a beginner; too difficult for anyone right now.

I would suggest looking around; if you want to learn about deep learning and RNNs, there are lots of tutorials on deep learning and RNNs. I suggest starting with something more standard as your first project, until you get a better handle on what kinds of tasks are within reach of current methods.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ That's actually about what I expected to be the answer to the problem, after all, if that power existed, I would expect many weird books coming out and a lot of media fuss. But it's a bit disappointing to see that not that much progress since 2015 (when I last checked it) hasn't been made. $\endgroup$ – Philippe Fanaro Jul 28 '18 at 0:23
  • $\begingroup$ Still, do you have a recommendation of a good book for a beginner with some mathematical background? $\endgroup$ – Philippe Fanaro Jul 28 '18 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.