2
$\begingroup$

I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse 16, the marker will say 16-18. I wonder if, given I have a lot of other versions that separate them discretely, I can train an NLP model to separate those combined verses into discrete verses. I'm fairly new at deep learning, having done a few toy projects. I wonder how to think about this problem? What kind of problem is it? I think it might be similar to auto-punctuation problems and it seems the options there are seq2seq and classifier. This makes more sense to me as a classification problem, but maybe my inexperience is what drives me that direction. Can people suggest ways to think about this problem and resources I might use?

In answer to questions in the comment, I am dealing only with text, not images. An example might be like this:

Genesis 2, New Revised Standard Version:

5 when no plant of the field was yet in the earth and no herb of the field had yet sprung up—for the Lord God had not caused it to rain upon the earth, and there was no one to till the ground; 6 but a stream would rise from the earth, and water the whole face of the ground— 7 then the Lord God formed man from the dust of the ground, and breathed into his nostrils the breath of life; and the man became a living being.

Genesis 2, The message version:

5-7 At the time God made Earth and Heaven, before any grasses or shrubs had sprouted from the ground—God hadn’t yet sent rain on Earth, nor was there anyone around to work the ground (the whole Earth was watered by underground springs)—God formed Man out of dirt from the ground and blew into his nostrils the breath of life. The Man came alive—a living soul!

The goal then would be to divide the message version into discrete verses in the way that the NRSV is. Certainly, a part of the guide would be that a verse always ends in some kind of punctuation, though while necessary it is not sufficient to assign a distinct verse.

$\endgroup$
  • 2
    $\begingroup$ Is the idea to use an existing translation (or possibly multiple existing translations) to guide the splitting? I suspect that your problem may be unsolvable in general. The reason why the verses were combined in the first place was probably to allow for a more dynamic and idiomatic translation into English, which may mean moving words or even whole clauses around. Languages such as Greek (which has noun declension) or Hebrew (which has the definite direct object marker) have a more "free" word order than subject-verb-object languages such as English. $\endgroup$ – Pseudonym Sep 14 at 1:56
  • $\begingroup$ Interesting problem, by the way! You may want to ask this on Artificial Intelligence and tag it with natural-language-processing. $\endgroup$ – Pseudonym Sep 14 at 1:58
  • $\begingroup$ @Pseudonym that's right, I have about 30 versions that could act as a training set. Thanks for the sugestion, I'll post it over at artificial intelligence and see what I get. $\endgroup$ – rwreed Sep 14 at 13:16
2
$\begingroup$

In my opinion, this may be best approached as a sequence tagging problem, similar to part-of-speech tagging or named entity recognition. (So, this would be the seq2seq option, rather than regular classification). For example, think about it as trying to decide for each token, whether it is the start of a new verse or not.

The advantage of the sequence tagging approach compared to regular classification (at least when using some kind of neural architecture) is that you do not have to predetermine the length of the context taken into account for each distinct tagging decision.

To be clear, the classification approach (e.g, take 20 tokens to the left and 20 tokens to the right and decide whether the middle token is a verse boundary) is also viable, but the sequence tagging approach seems the more natural choice for me.

A final thought on the problem: Are you sure that (any kind of) classification/tagging approach is the right place to start? You may also want to think about this in terms of an alignment problem where you try to figure out which sentences from message edition correspondent to the ones in the NRSV version. Verse segmentation then comes for free. Alignment problems have been studied extensively in statistical machine translation.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.