I'm currently working on a personal software project (as a hobby) and would like to know which kind of approach I should follow according to you.
My application scans some websites (like Amazon or eBay) to check items of a specific category and show them all together ordered, for instance, by price.
Currently I use Python with LXML to extract strings and numbers from specific positions on the page (searching by HTML element tag or CSS class).
This approach works smoothly, but I'm a little bit concerned about scalability: what if I want to add more web sources? What if Amazon and eBay change the structure of the HTML page?
So far I managed these issues by manually changing the functions I use to retrieve data from each website, but was wondering if there is a better approach, for example using AI/Machine Learning.
Finally the question: is it better to rely on "old-style" and "hard-coded" algorithms for such a scenario or Machine Learning and AI could be more helpful according to you?
Thanks a lot!