I'm working on a "semantic" browser engine where all search engines should look the same. One way to do this is to hard-code parsing rules for each site; another is to use machine-learning. Of course the latter alternative is the hip and cool one. ;)
I need a method to cluster links from search results from Bing, Google and other popular search engines. Menu links, pagination, etc, should be in a different cluster than listed search results. I've tried KMeans, it didn't work so good. DBSCAN worked fine with Google, but I couldn't easily transfer the result to Bing.
There are a lot of different features one can consider for HTML elements, in my case
<a>, anchors. I've tried combining number of children (recursively), text length, number of attributes and some more. Is it possible to advice in which direction to go? I have no former experience with machine-learning, so tuning an algorithm by hand like this is not trivial.
Thankful for any feedback.