Metadata
- Author: Variance Explained
- Full Title:: Is Bayesian AB Testing Immune to Peeking. Not Exactly
- Category:: 🗞️Articles
- Document Tags:: Bayesian testing
- URL:: http://varianceexplained.org/r/bayesian-ab-testing/
- Finished date:: 2024-05-04
Highlights
Bayesian statistics are useful in experimental contexts because you can stop a test whenever you please and the results will still be valid. (In other words, it is immune to the “peeking” problem described in my previous article). (View Highlight)
The claim of “Bayesian testing is unaffected by early stopping” is simply too strong. Stopping a Bayesian test early makes it more likely you’ll accept a null or negative result, just like in frequentist testing. And being overconfident in a statistical method is often a much greater danger than any flaws in the method itself. (View Highlight)
we’re concerned about the effect of focusing on the expected loss, and thereby letting features that offer no improvement pass the test. (View Highlight)
methods. P-values can be counter-intuitive, yes. But switching to Bayesian testing requires getting everyone familiar with priors, posteriors, and the loss function- it’s trading one set of challenging concepts for another (View Highlight)
Alternative Bayesian procedures. For example, focusing just on the expected loss may lead us to call null results significant. But if we base our procedure not just on the expected loss, but also the posterior probability Pr(B>A)Pr(B>A), we can focus on cases where we’re confident in a positive change (View Highlight)