4
$\begingroup$

I'm interested to see if I can use machine learning/network analysis methods to automatically detect automatically generated (spam) webpages. I'm particularly interested in webpages that look structurally like a non-spam website, but on closer inspection are total rubbish.

If I want to test a method, I'd need some way of accessing (or generating) these webpages.

Question: Where can I access automatically generated (spam) webpages?

Here's a sample of the kind of webpage I'm thinking about:

Sample spam Image source

$\endgroup$
  • $\begingroup$ This one looks like it uses a Markov chain, together with a pre-generated template. See en.wikipedia.org/wiki/Markov_chain#Markov_text_generators for references. $\endgroup$ – András Salamon Jun 27 '13 at 16:44
  • $\begingroup$ This is barely ontopic, if at all. Community votes, please: is this a valid machine-learning question, or a request to Google for the OP? $\endgroup$ – Raphael Jun 20 '16 at 12:15
3
$\begingroup$

Question: Where can I access automatically generated (spam) webpages?

You could use:

  • a parody generator program.

    They're usually based on Markov chains (e.g. SubredditSimulator) or context-free grammars (e.g. the well known SCIGen... according to the authors "with Markov chains you might get something syntactically correct, but it is likely to be boring"!).

    Also a cut-up technique could be enough.

  • a (web spam) dataset. The WEBSPAM-UK2007 collection is one of the largest and cleanest (and it's open). A description of the dataset can be found in A Reference Collection for Web Spam (the paper describes WEBSPAM-UK2006 but it's still valid).

    You should check if it's too broad: the collections includes many different web spam aspects as possible (pages that are only advertising, pages with unrelated links, pages that automatically redirect...) and you seems only interested in pages that contain machine generated content.

  • the spam messages in the email: email spammers often include URLs in their spam messages. Following those links you can store the HTML content. You should add a few heuristics to identify some false positives (e.g. exclude Alexa's Top 500 list and use SiteAdvisor's rating system to compile a whitelist).

    This is the method used to create another well known dataset: Webb Spam Corpus (the original corpus was collected by Steve Webb). Further details are contained in Introducing the Webb Spam Corpus: Using Email Spam to Identify Web Spam Automatically (Steve Webb, Jamesa Caverlee, Calton Pu).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.