I work on a web project that handles region based user submissions. We currently have about 50 regions, some receive a large number of submissions, some receive next to none.

We also ingest case data for these submissions from a system, again some regions may supply lots of data, others not so much.

The number of cases ingested and submissions will differ during a weekday and a weekend, and holiday periods (e.g Christmas) will be quieter.

I'm wondering what algorithm I can use to detect and alert on an unexpected variance in the data? E.g if we've been receiving 100 submissions for region X each day, then suddenly that number halves. But that number may be low anyway during the weekend.

I was thinking about possibly alerting on the intersection of two moving averages - maybe a 7 day and a three day, but that might be too slow / unresponsive. Another option may be detecting a %age variance based on the figures for the same day for the past few weeks. What is the best solution?

  • $\begingroup$ Step 1: define "unexpected". There infinitely many possibilities -- use any form of regression and any distance measure -- so you'll have to specify and/or tinker more. $\endgroup$ – Raphael Jul 7 '16 at 10:43
  • $\begingroup$ 1. I suggest you start by identifying what factors you think might legitimately influence the number of submissions (i.e., in a way that you don't want to generate an influence). Then, figure out if you can measure those factors. Then, edit your question to explain. You say "e.g., weekend", but are there others? Try to list or summarize all. 2. Take a look at en.wikipedia.org/wiki/Bollinger_Bands and en.wikipedia.org/wiki/Moving_average. $\endgroup$ – D.W. Jul 7 '16 at 18:56
  • $\begingroup$ 3. The question is fine here and on-topic here, but you might get more sophisticated answers on Stats.SE. Don't re-post your question there, but if you'd prefer to see your question there, you can flag it for moderator attention and we can migrate it. $\endgroup$ – D.W. Jul 7 '16 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.