I'm working on a "semantic" browser engine where all search engines should look the same. One way to do this is to hard-code parsing rules for each site; another is to use machine-learning. Of course the latter alternative is the hip and cool one. ;)

I need a method to cluster links from search results from Bing, Google and other popular search engines. Menu links, pagination, etc, should be in a different cluster than listed search results. I've tried KMeans, it didn't work so good. DBSCAN worked fine with Google, but I couldn't easily transfer the result to Bing.

There are a lot of different features one can consider for HTML elements, in my case <a>, anchors. I've tried combining number of children (recursively), text length, number of attributes and some more. Is it possible to advice in which direction to go? I have no former experience with machine-learning, so tuning an algorithm by hand like this is not trivial.

Thankful for any feedback.

  • 1
    $\begingroup$ Hard-coded parsing is definitely the way to go. You use machine learning only when simple solutions don't work. $\endgroup$ – Yuval Filmus Sep 10 at 22:25
  • $\begingroup$ @YuvalFilmus Thank you for your comment. If I hard-code, I can only support a limited number of search engines. Machine learning has the potential to support all (assuming their way to display search result are similar). $\endgroup$ – Olle Härstedt Sep 11 at 5:58
  • $\begingroup$ Machine learning requires annotated datasets. You will need to provide such a dataset for each search engine you support. $\endgroup$ – Yuval Filmus Sep 11 at 6:57
  • $\begingroup$ @YuvalFilmus To train the algorithm? DBSCAN is an unsupervised algorithm. $\endgroup$ – Olle Härstedt Sep 11 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.