I am interested in experimenting with machine learning on the ASTs of programming langauges (e.g. some Java code). I would like to develop an algorithm that determines if a given AST satisfies some predicate. I have a training set of programs that do and do not satify the predicate.
The machine learning techniques that I am aware of operate on vectors or matricies. Are there techniques that operate on trees?
Your problem can be solved by existing machine learning methods. So, instead of searching for new technique you should choose from existing techniques. The main part of your problem which separates it from general machine learning problems is AST data which will act as input to your algorithm.
There are a lot of feature extraction techniques available which will help you convert AST data into usable format. I would suggest doing tree traversals to generate a feature vector from your AST and then applying machine learning techniques on it.
Probably due to recent popularity (and relative accessibility) of certain algorithms, like neural networks, we often think of "machine learning" as being about heavy statistical/numerical algorithms. The Machine Learning Wikipedia page mentions that machine learning is often conflated with data mining. As that page demonstrates, there are other facets to it. In particular, there's the "programming-by-example" community with one of the tools being inductive logic programming (ILP) or inductive programming in general.
You should browse the Machine Learning Wikipedia page for other algorithms, and I'm not suggesting that you couldn't apply a more numerical algorithm like neural networks to this problem, but at least some forms of ILP seem like they would be a good fit. Essentially the goal there is to learn a logic program that correctly classifies examples given positive and negative examples. A logic program is just a collection of predicates.
