Metadata
- Author: Evan Miller
- Full Title:: How Not to Run an a/B Test
- Category:: 🗞️Articles
- Document Tags:: Hypothesis testing,
- URL:: http://www.evanmiller.org/how-not-to-run-an-ab-test.html
- Finished date:: 2024-05-04
Highlights
Since you are going to fix the sample size in advance, what sample size should you use? This formula is a good rule of thumb: Where is the minimum effect you wish to detect and is the sample variance you expect. Of course you might not know the variance, but if it’s just a binomial proportion you’re calculating (e.g. a percent conversion rate) the variance is given by: (View Highlight)
UPDATE, May 2013: You can see this formula in action with my new interactive Sample Size Calculator. Enter the effect size you wish to detect, set the power and significance levels, and you’ll get an easy-to-read number telling you the sample size you need. END OF UPDATE (View Highlight)
report how large of an effect can be detected given the current sample size. (View Highlight)