I'd like to measure how different two HTML files are from each other.

I refer to the following scenario:

  1. User A visits a website, let's call the website example.com
  2. User A shares the website with User B
  3. User B visits the shared website
  4. User A and User B have installed an add-on that each sends the visited web pages as HTML files to a server C.
  5. The server C should evaluate to what extent the website differs from User A and User B.

I would like to measure the degree of individualization of a website onto the user.

Some websites adapt to each user very strongly, for example the Netflix homepage. Other pages look the same for every user, for example the Terms and Conditions page of Netflix.

My questions are now:

  1. Are there any research or algorithms for this problem?
  2. Which algorithms can be used for this kind of problem?

My thoughts:

  1. http://www.php.net/manual/en/function.similar-text.php
  2. Levenshtein distance
  3. Jaro-Winkler distance
  4. Anything with AI
  • 2
    $\begingroup$ diff and bsdiff algos, then measure the output size $\endgroup$ – Bulat Sep 10 '18 at 21:10
  • $\begingroup$ another approach - compute how much 16-byte strings from the first page appears in the second one. it should be trivial using suffix array. $\endgroup$ – Bulat Sep 10 '18 at 21:12
  • 2
    $\begingroup$ There are many possible distance metrics you could use. You've listed some reasonable ones. There's no way to choose among them from the information provided here, as you haven't listed any requirements or criteria that such a metric must satisfy. $\endgroup$ – D.W. Sep 11 '18 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.