Finding (and possibly extracting) source code in heterogenous text data set

Question

I'm looking for a way to recognize and possibly extract source code from text files that may contain only source code, source code mixed with plain text or just plain text without any source code.

There's different source code contained in the data set, just to name a few examples: Java, C, Python, Bash, PHP, Javascript, ...

An example of a mixed file would look something like this:

Title
=====

Introduction
------------
dolor sit amet, consetetur sadipscing elitr, sed diam
nonumy eirmod tempor invidunt ut...

Description
-----------
Labore et dolore magna aliquyam erat, sed diam voluptua:

    [.c snippet]

At vero eos et accusam et justo duo dolores et ea rebum.

Proof of Concept
----------------
Stet clita kasd gubergren, no sea takimata sanctus est
Lorem ipsum dolor sit amet.

-- sourcefile.c --

[Full .c source file]

The above is only an example and while there might be some files with similar layout, most of them differ quite a bit.

Only looking for one specific language would be preferred, but not required.

My first naive approach when looking for Java source code was to run a simple keyword search using Java keywords, but the results weren't that great. Most java keywords are neither unique to java nor sparingly used in plain text, so what turned up as top results was mostly C files, and of course, this doesn't tackle extraction at all.

I recognize that this is a difficult problem, and I've had no success googling for this topic. The most interesting thing I've found was this paper titled Extracting Code Segments and Their Descriptions from Research Articles, but it's not quite relevant since it's much more focused on the Descriptions part than it is on the Code Segments part.

Is there some kind of algorithm or existing program or library that solves this or a similar problem? Or maybe some kind of neural network flavor?

Answer 1

There are many possible ways you might do this. I'll suggest one.

I suggest you train a classifier to recognize whether a sequence of characters is code or text. Here, the goal is to build a boolean classifier that either outputs "text" or "code". You'll train it only on documents that are entirely text or entirely code (no mixed documents). Build a classifier that not only outputs a classification, but also a confidence (a likelihood value for the chosen classification).

Now, to apply it to mixed documents, run the classifier on all substrings. If a substring $S$ is classified as "code" with high confidence (high likelihood), above some threshold, and there's no superstring $S'$ of $S$ (of length not too much longer than $S$'s length) that is also classified as "code", then output $S$ as one of the code elements from the mixed document. You'll need to select a threshold with a little bit of experimentation.

I suspect this might work nicely. You'll have to try it to see, though.

So, how do you build such a boolean classifier? One simple approach would be to use a simple linear classifier (e.g., SVM or naive Bayes) with features based on bag-of-words on lexical tokens. That alone might suffice (e.g., code tends to have many instances of tokens like "{" and "return", but text doesn't). You could also try bag-of-words on individual characters. If you want to build a more sophisticated classifier, you could train a character-level recurrent neural network (e.g., a LSTM or a CNN); this might achieve higher accuracy, given a large enough training set, but probably will also require a bit more technical knowledge.