I am a statistician by training, so am not too familiar with the computer science literature. One of the engineers at the firm I work at told me about problems with A/B testing in Canary releases.
I am well versed with the literature on A/B testing, but in the case of canary release, I am not quite sure what is being tested. The load a service gets over time is going to vary, so using naive tests where you assume independence of some form is not going to make the cut. My particular questions are
1) What is being tested? What are the assumptions about the data
2) What are "the metrics" which people keep mentioning in the blog articles on the subject and
3) In practice, when does an engg. say that systems look in a dangerous state? Does this mean that you see larger fluctuations OR any time you see a very big fluctuation which exceeds the normal expectations?
4) After how much time do you typically decide that you are satisfied with the results, and should deploy to the other nodes?
I am looking for a formal problem setup if possible. Could someone please suggest starting points for me to look at