Easy Evaluation of Decision Rules in Bayesian aB Testing

Metadata

• Author: chrisstucchio.com
• Full Title:: Easy Evaluation of Decision Rules in Bayesian a/B Testing
• Category:: 🗞️Articles
• Document Tags:: Bayesian testing,
• URL:: https://www.chrisstucchio.com/blog/2014/bayesian_ab_decision_rule.html
• Finished date:: 2024-05-04

Highlights

The basic idea of the decision rule is as follows.

1. Choose a “threshold of caring” - if A and B differ by less than this threshold, you don’t care which one you choose.
2. Choose a prior on the distribution of conversion rates of A and B.
3. Compute a posterior, and use it to estimate whether the expected losses you’d make by choosing A (or B) are below the threshold of caring. If so, stop the test (View Highlight)