I am a statistician by training, so am not too familiar with the computer science literature. One of the engineers at the firm I work at told me about problems with A/B testing in Canary releases.

I am well versed with the literature on A/B testing, but in the case of canary release, I am not quite sure what is being tested. The load a service gets over time is going to vary, so using naive tests where you assume independence of some form is not going to make the cut. My particular questions are

1) What is being tested? What are the assumptions about the data

2) What are "the metrics" which people keep mentioning in the blog articles on the subject and

3) In practice, when does an engg. say that systems look in a dangerous state? Does this mean that you see larger fluctuations OR any time you see a very big fluctuation which exceeds the normal expectations?

4) After how much time do you typically decide that you are satisfied with the results, and should deploy to the other nodes?

I am looking for a formal problem setup if possible. Could someone please suggest starting points for me to look at

  • $\begingroup$ "incremental rollout" or "phased rollout" would help in the search. Maybe something like this? Also it is a "reference request" and I am not sure what you are looking for. $\endgroup$ – Evil Oct 14 '17 at 0:35
  • $\begingroup$ On what subject? There's lots written about A/B testing. Have you done any research and reading before asking? What do you currently understand about A/B testing, and what specifically are you confused about? What's a "canary release"? Is there any reason to think that A/B testing of "canary releases" is any different than A/B testing of anything else? What are the problems that the engineer was referring to? $\endgroup$ – D.W. Oct 14 '17 at 2:49
  • $\begingroup$ My apologies for the vagueness. I have edited the question to the best of my knowledge. From what I understood, this seems to be a generic problem which is encountered across multiple firms. If the question still makes no sense, I will delete it $\endgroup$ – Sid Oct 14 '17 at 3:49
  • $\begingroup$ I don't think this question is answerable in its current form. I think you need to ask the person who told you that, not us here. I don't think we can tell you what is being tested. You'll need to ask that of the people who are doing the testing. There are many things one could test (e.g., reliability, crashes, user activities), but it's up to the people doing the testing what they are actually measuring. Similarly for "the metrics"; there are many metrics; there's not a single metric that's always used. $\endgroup$ – D.W. Oct 14 '17 at 4:46
  • 1
    $\begingroup$ Not sure if this fits better here or on Software Engineering. $\endgroup$ – Raphael Oct 14 '17 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.