how to identify patterns in program execution flows?

Consider a library, that is used by different programs. Under certain (unknown) circumstances, an error can occur. I want to feed execution traces of faulty programs to some kind of an algorithm / framework so it learns about the "pattern" which led to the error. It should then, when fed a live trace, detect when that error will occur and halt the program execution.

So what I have is a list of totally ordered invocations of library functions with their parameters and return value. A simple fault program, where a memory area is used after being freed, may look like

mylib_allocate()=0x100
mylib_dosomething() // unrelated "noise"
mylib_free(0x100)
mylib_dosomething() // unrelated "noise"
mylib_set(0x100, 0); // error here


So my question is: Are there any existing frameworks that could be fed with that kind of data, which tries to automatically identify patterns from many traces leading to the same error (e.g. "use after free"), and which produces some output which could be used to detect the error (i.e. trigger an event if such an error is going to happen)?

I looked through some data mining approaches, but it seems like they all target "manual" analysis for pattern identification, e.g. by graphic processing to visualize potential "cluster". Then I tried to look at "process mining", but this seemed to be a dead-end as well, because - as far as I understand - my invocation data with an arbitary amount of parameters doesn't fit into that "process" schema. At least I didn't get any reasonable results with ProM.

• ProM is an approach and a framework rather than a single tool. It has hundreds of plugins using very different techniques and suited for very different problems. There may be a plugin suited to your needs. – reinierpost Jan 9 '17 at 11:15

4. Also, you might need to do some feature engineering. For instance, when you have pointers or handles (like 0x100), you might want to canonicalize them, as the only useful information is whether two calls used the same pointer or a different pointer -- the actual value of the pointer is irrelevant. Since you have domain knowledge of that fact, by transforming the data before you feed it to the data mining / machine learning framework, you can help it learn a lot faster and more effectively.