# How can I express the similarity between a Bing and a Google search result?

I'm working on a "semantic" browser engine where all search engines should look the same. One way to do this is to hard-code parsing rules for each site; another is to use machine-learning. Of course the latter alternative is the hip and cool one. ;)

I need a method to cluster links from search results from Bing, Google and other popular search engines. Menu links, pagination, etc, should be in a different cluster than listed search results. I've tried KMeans, it didn't work so good. DBSCAN worked fine with Google, but I couldn't easily transfer the result to Bing.

There are a lot of different features one can consider for HTML elements, in my case <a>, anchors. I've tried combining number of children (recursively), text length, number of attributes and some more. Is it possible to advice in which direction to go? I have no former experience with machine-learning, so tuning an algorithm by hand like this is not trivial.

Thankful for any feedback.

• Hard-coded parsing is definitely the way to go. You use machine learning only when simple solutions don't work. – Yuval Filmus Sep 10 at 22:25
• @YuvalFilmus Thank you for your comment. If I hard-code, I can only support a limited number of search engines. Machine learning has the potential to support all (assuming their way to display search result are similar). – Olle Härstedt Sep 11 at 5:58
• Machine learning requires annotated datasets. You will need to provide such a dataset for each search engine you support. – Yuval Filmus Sep 11 at 6:57
• @YuvalFilmus To train the algorithm? DBSCAN is an unsupervised algorithm. – Olle Härstedt Sep 11 at 10:48