Dr. Mario's 2nd 🧠

Rapid and Scalable Bayesian AB Testing

3 min read

rw-book-cover

Metadata

Highlights

Statistical power in large-scale tests – as the number of categorical factors and possible values of these factors grows, the amount of traffic allocated to each combination of values reduces. Hence a t-test run separately for each such combination would suffer from reduced statistical power, and consequently require the test to be run for a longer duration before true differences can be detected (View Highlight)

Hierarchical Bayesian inference is a well-established methodology previously articulated by Gelman and others [7], [8]. (View Highlight)

rw-book-cover

Metadata

Highlights

the true but unknown probability of a response is represented as rf, where the vector f = (m, c) is an element of F = M× C and denotes a specific content m presented to the user with a particular context c. Given a total of F = M + C content and context factors, each with cardinality of at most N, the number of unique f grows exponentially as O(NF). One of the statistical challenges with multivariate AB testing is that, as N and F grow, it becomes increasingly challenging to estimate each rf with sufficient statistical power. Our hierarchical Bayesian approach ameliorates this challenge, by pooling knowledge across different instances of f to increase statistical power. (View Highlight)

we use a Bayesian hypothesis testing framework that builds on the mSPRT [4]. The framework sequentially evaluates the relative evidence that there is a statistically significant difference between pairs of Bayesian estimates ˆrh f s – all the while maintaining statistical validity without the need for post hoc corrections. (View Highlight)

Another perspective is that experiments can be characterised by a set of common features. Within this perspective, pooling information and learnings across experiments can help experimenters build robust intuition as to the sorts of features that yield most impactful experiments. (View Highlight)

Hierarchical Bayesian inference, in addition to being useful for modelling the variables within an experiment, can also be used to conduct such meta-analysis across experiments. By encoding distributional assumptions, we can learn the latent hyperparameters that can explain the impact of different experimental interventions. (View Highlight)

Graph View