Metadata
- Author: Olga Berezovsky
- Full Title:: How to Run an a/B Testing on Low Traffic - Issue 181
- Category:: 🗞️Articles
- Document Tags:: Hypothesis testing,
- URL:: https://dataanalysis.substack.com/p/how-to-run-an-ab-testing-on-low-traffic?utm_source=substack&utm_medium=email&utm_campaign=posts-open-in-app&triedRedirect=true
- Finished date:: 2024-10-05
Highlights
I believe A/B testing becomes worthwhile when: • (a) you consistently have 100+ users in a given day (then you could scope for X weeks), AND • (b) your traffic volume remains relatively stable without significant fluctuations (then you could estimate MDE). (View Highlight)
Make sure the variance is the same for both Control and Variance, especially for user tenure. (View Highlight)
Users might initially respond with high interest (high CVR) that could “normalize” over time. They might also do the opposite, instead of resisting the change initially (low CVR) before accepting it. This is reflected in the Variant showing unusually high or low CVR, which was not expected in your MDE. (View Highlight)