Users can be assigned to one experiment on my site. I have an API that developers use to trigger logic for each experiment. They call ExperimentEngine.run() to trigger the code logic for the experiment.
I would like to allocate users to each experiment, at the point where a user might be exposed to the logic for that experiment. I would like to assign users to experiments such that experiments implemented downstream (on pages that are usually seen last) still get users assigned to them.
For example, if user A is runs into the code logic for experiment A at login and then goes to page B and runs into experiment B, the user A could be assigned to either experiment A or B. That means that they will only see one of the experiments and not both (either A or B) or neither. I would like to figure out the right algorithm so that experiment B (which is downstream and shown to the user after they've seen experiment A) gets users assigned to it. I don't to assign all users to experiment A.
So the flow is as follow
- User visits page A where experiment A is implemented
- We decide whether to assign users to experiment A. If user is assigned to A, user will be able to see experiment A.
- User visits page B where experiment B is implemented, we decide whether to assign users to experiment B
- Users can only see experiments that they are assigned to.
- I want to come up with an algorithm that allows me to assign users to experiments regardless of where on the site they are implemented so that the traffic distribution is efficient and experiments implemented downstream get enough users.
Can someone please point me in the right direction to an algorithm that allows me to do the above?
Each experiment is given X number of people needed per day to reach stats sig. We could factor that into the algorithm to make sure that we assign users to experiments in a way that experiments reach their sample size in two weeks.
A possible algorithm:
- For each experiment, we make a decision of whether to assign based on the experiment's location using coin flip.
- If we get heads, a list of experiments that are implemented for that location are selected.
- An experiment is chosen from that list based on priority system. At every location, a % of users are assigned to one of the experiments implemented at that location.
- When we decide to assign or not to assign to any experiments at that location, that decision is not made again for the user.
The Question: How do I assign users to one experiment at a time so that experiments that they might see last get enough users? Only users assigned to experiments can see them.