![]() That’s good news for any peekers who are willing to risk the occasional flat result or slight decrease to take advantage of shorter run times while avoiding any large declines. The likelihood of getting a false positive when there’s even a modest drop in conversion rate is not very high in later stages of the test. We see that peeking late in the game-even intermittently-may more than triple the number of false positives, but that’s on a very small base. But, by and large, are testers who peek making bad decisions? I wanted to know, so I simulated thousands of tests to find out. Much has been said on this blog and elsewhere about the evils of peeking. Many experimenters, however, violate this assumption, peeking at the stats while the test is running and abandoning tests early when results are convincing-introducing unquantified error into their decision-making. The classic breed of A/B testing methods-known as fixed-horizon tests-assumes that a specific sample size has been committed to in advance the statistics rely on this assumption for validity. Sacrificing relationship capital with stakeholders eager to make a decision.Delaying the monetization of a valuable change.Prolonging the exposure of a poor experience.Forgoing the launch of other experiments. ![]() ![]() Time is a cost of running a test, but, in practice, it also represents the cost of continuing a test: In my experience, this is a dilemma almost every experimenter faces at one point or another-regardless of company size or traffic count. If your arm is waving vigorously in the air right now, this post is for you. Raise your hand if you’ve ever struggled with a decision between disciplined testing procedures and expedient decision-making.įor example, think of a time when you’ve had to decide between sticking to your A/B test design-namely, the prescribed sample size-and making a decision using what appears to be obvious, or at least very telling, test data. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |