# Finding (and possibly extracting) source code in heterogenous text data set

I'm looking for a way to recognize and possibly extract source code from text files that may contain only source code, source code mixed with plain text or just plain text without any source code.

There's different source code contained in the data set, just to name a few examples: Java, C, Python, Bash, PHP, Javascript, ...

An example of a mixed file would look something like this:

Title
=====

Introduction
------------
dolor sit amet, consetetur sadipscing elitr, sed diam
nonumy eirmod tempor invidunt ut...

Description
-----------
Labore et dolore magna aliquyam erat, sed diam voluptua:

[.c snippet]

At vero eos et accusam et justo duo dolores et ea rebum.

Proof of Concept
----------------
Stet clita kasd gubergren, no sea takimata sanctus est
Lorem ipsum dolor sit amet.

-- sourcefile.c --

[Full .c source file]


The above is only an example and while there might be some files with similar layout, most of them differ quite a bit.

Only looking for one specific language would be preferred, but not required.

My first naive approach when looking for Java source code was to run a simple keyword search using Java keywords, but the results weren't that great. Most java keywords are neither unique to java nor sparingly used in plain text, so what turned up as top results was mostly C files, and of course, this doesn't tackle extraction at all.

I recognize that this is a difficult problem, and I've had no success googling for this topic. The most interesting thing I've found was this paper titled Extracting Code Segments and Their Descriptions from Research Articles, but it's not quite relevant since it's much more focused on the Descriptions part than it is on the Code Segments part.

Is there some kind of algorithm or existing program or library that solves this or a similar problem? Or maybe some kind of neural network flavor?

• You might look up the source code for the file Unix command which will report the file type for a file of unknown origin. – user1118321 Feb 18 '18 at 2:02

Now, to apply it to mixed documents, run the classifier on all substrings. If a substring $S$ is classified as "code" with high confidence (high likelihood), above some threshold, and there's no superstring $S'$ of $S$ (of length not too much longer than $S$'s length) that is also classified as "code", then output $S$ as one of the code elements from the mixed document. You'll need to select a threshold with a little bit of experimentation.