0
$\begingroup$

We're accessing an API of a web system for obtaining product information. We require some additional information, which is not available through the API. This information is publically available for each product visually - through the source code for each item on its item page. We've written an algorithm which parses this information from the web page for each item, but that will be highly ineffective in the long run, since the algorithm will simply stop working if and when they decide to change the source code ( for example - redesign of the front end ).

I feel like there should be a supservised learning approach to this problem, however I'm unaware if there exist such solutions.

What are some good aproaches to this kind of problems ?

Regards.

Improve this question
$\endgroup$
2
  • $\begingroup$ I think if you were not able to get this information as a raw data through the API, then you simply can't have it in the long run. I am not sure even if ML approaches will help. $\endgroup$
    – seteropere
    Commented Mar 9, 2014 at 18:19
  • 1
    $\begingroup$ You would need to add some actual detail about your problem. What is "the information?" What is available? How are they related? Do you want to build something that maps from "information available through the API" -> "the information" or "web page source code" -> "the information"? If the later, my guess is you would be better off coming up with some heuristics, but again, it depends on what you're actually trying to do. $\endgroup$
    – alto
    Commented Mar 10, 2014 at 18:40

2 Answers 2

Reset to default
2
$\begingroup$

In short, this is probably difficult.

Your question is not fully specified, since you did not write the details of the data you are parsing and since estimating the possible changes to it is a matter of guesswork.

Once this web site changes format, then if we assume that the products remain the same, you are essentially looking at supervised learning task: given a list of known product web pages, make an "extractor function" that outputs the desired field.

If you are lucky and all the product pages are identically structured then this problem might be quite easy. You simply need to identify the path to the node of the HTML in which the desired information lies. On the other hand, if the web pages have a varied structure, or no structure at all, then the problem may be much more difficult.

There are some academic works in these areas, but don't get your hopes up. Look for the keywords "automatic web extraction".

Improve this answer
$\endgroup$
1
$\begingroup$

You are crawling the web for some structured information inside the web pages.

There is an infinite number of ways to represent the information in a web page (like java scripts); it means that it's impossible to have an automated system to do that for you completely. I think, this process must be handled by a person up to a level.

My suggestion is this:

  1. Once you finished your parser,
  2. select a sample page (one of the product pages) for testing whether that structure remained the same or not,
  3. once in a while, try to check if the structure is changed,
  4. if your program realized that structure is changed, then it can try to learn the pattern by looking up the information, or if it wasn't successful it can send you some feedback about the structure that is changed and you must handle it via new parser.

if the products are changing and you cannot select a sample product to check the structure, I suggest:

  1. Write a regular expression for the pattern of information in the web page,
  2. Check if the regular expression returns anything or it's not valid anymore,
  3. If the regular expression is not valid, then the structure is changed and you need to come up with a new one
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.