# How to represent text for a program to add punctuation to a block of text?

I want to try and make a program where I remove the punctuation from a block of text and it then inputs it in the correct places. I have started with a simpler element of just using text and spaces, no commas, full stops etc. The first angle I have been taking this is with machine learning, neural networks and simpler word count probabilities. However the thing I am struggling with is the representation of the block of text. At the moment I give it a block of text and it reads say 20 letters then breaks it into test based on where it thinks a word is. This is ineffective though because it may just cut off half a word at the start/end of the block. Just wondering if anyone had an idea of a way to use the block of text in this way in a better way to do this task! Many thanks

I am making two assumptions here about things that, to me, are not quite clear from your question

• When you say you removed punctuation, you actually meant whitespaces as well.
• You tried neural networks, but not in a sequence-to-sequence fashion.

What's nice about your problem is that you can generate training data as much as you want just by taking a proper piece of text and removing all the punctuation.

This being said, I suggest you engage your problem as a machine translation task where you want to predict the proper text given the one without punctuation.

Today, the best way to do this is by sequence-to-sequence learning with deep neural network (just pick a few machine translation papers from last ACL (= Annual Meeting of the Association for Computational Linguistics) conference to learn more about that.

Concerning you actual question, how to represent the input, I'm not quite sure if you have to limit the size of your input beforehand or not. In any way, you could choose the size of your input window so that any block of text that you possibly want to put in would fit (if you know beforehand how long your paragraphs will be). Or it might be that you can just cut your input sequence at an arbitrary letter and your model will learn to make sense of it anyway (I am not an expert in neural machine translation).

So, as you can see, my answer is maybe not completely on the spot but I hope it is helpful to you anyway.

Before each space, you have a decision to make: do you add some punctuation or not? And if yes, what kind of punctuation? (comma? period? semicolon?)

I suggest you treat that decision as the machine learning task. Build one boolean classifier per type of punctuation. Classifier #1 answers: "should I insert a comma before this space?" Classifier #2 answers: "should I insert a period before this space?" etc.

Next, choose some features that you think might be helpful in making this prediction/decision. Here are some candidates to get you started and get you thinking:

• Is the next letter after the space capitalized or not?

• Is the previous word before the space capitalized or not?

• Is the previous word before the space an abbreviation of a title? (Mr, Mrs, Dr, etc.; you can pretty easily build a small list/dictionary of such words)

• Is the next word after the space part of a proper noun? (a part-of-speech tagger might be useful here; see also entity resolution)

• Is the next word after the space one of the set of words that are always capitalized no matter where they appear in the sentence? (grep '^[A-Z]' /usr/share/dict/words will get you a list/dictionary of candidate such words; this is a boolean feature that tests whether the next word is in that list or not)

• What is the next word after the space? (word vector via some word embedding)

• What was the previous word before the space? (word vector)

• If you ran a NLP part-of-speech tagger and grammar parser on the part of the sentence up to this point, does it appear to be a complete sentence? (this gives a hint whether, if we were to put a period here, everything up to this point would be a complete sentence)

As you can see, I am suggesting you use a richer feature set, and one crafted for your particular task, rather than just using the previous 20 letters as the input -- and I am suggesting you use the spaces to help you identify potential insertion points, rather than potentially trying to insert punctuation between every pair of characters.