Retrieving data from website - Parser vs AI

Question

I'm currently working on a personal software project (as a hobby) and would like to know which kind of approach I should follow according to you.

My application scans some websites (like Amazon or eBay) to check items of a specific category and show them all together ordered, for instance, by price.

Currently I use Python with LXML to extract strings and numbers from specific positions on the page (searching by HTML element tag or CSS class).

This approach works smoothly, but I'm a little bit concerned about scalability: what if I want to add more web sources? What if Amazon and eBay change the structure of the HTML page?

So far I managed these issues by manually changing the functions I use to retrieve data from each website, but was wondering if there is a better approach, for example using AI/Machine Learning.

Finally the question: is it better to rely on "old-style" and "hard-coded" algorithms for such a scenario or Machine Learning and AI could be more helpful according to you?

Thanks a lot!

Answer 1

It depends what to use on what you are doing. I divide this problem in two categories.

1.Site without changing structure

Suppose, you are building a app which takes data from your national weather department site and normally which has same structure throughout decade. Here using AI for finding data is stupid thing. You should use conventional parsing technique here. and also your are getting data from number of sources but has fixed structure then you can go with this approach.

2.Site with changing structure

As per your example, I believe approach 1 is not so useful. because there are lots of site to parse and these site changes there structure periodically. So I believe using AI for parsing data from these sites is beneficial. I believe you used Optical Character Recognition. you will need similar functionality to your system. Your AI system first load page and see how it looks in browser (not code). Then finds appropriate fields from page using Named Entity Recognition from text which found using OCR

I think applying Named Entity Recognition to directly parsed code is sufficient to find different fields like item name, price, specifications, type etc

Answer 2

Zhou Mashuq have shown that you can learn the structure of documents and recognize them even with different templates. It does this by grouping content into labels. See diffbot.com for a commercial example of this.

While this recognizes the labels you still need to select the labels that interest you. This could be done by training a classifier, perhaps against your existing extracted values.

The effort of putting this together is several orders of magnitude higher than doing it manually (for some n<100 websites) and probably significantly less precise.