Metadata
- Author: discord.com
- Full Title:: Measuring Product Impact Without a/B Testing: How Discord Used the Synthetic Control Method for Voice Messages
- Category:: 🗞️Articles
- Document Tags:: Hypothesis testing,
- URL:: https://discord.com/blog/measuring-product-impact-without-a-b-testing-how-discord-used-the-synthetic-control-method-for-voice-messages?&aid=recx7eyNaCJXTDmss&_bhlid=086bb44e8d5bd6dcb5ebb740b4c6b5b27787fbb6
- Finished date:: 2024-11-30
Highlights
simple recipe: • Outcome data for one treatment unit, multiple time periods before and after treatment. • Outcome data for multiple control units, for those same time periods. • A trusted analytical library to generate the synthetic control counterfactual, e.g. the Synth package in R, or the SyntheticControls package in Python. You can evaluate how well your synthetic control has, well… controlled… by examining the Mean Squared Prediction Error (MSPE) before and after the feature was rolled out. If your MSPE gets much bigger after the feature rolls out, that means your treatment effect is working properly! You can also do a sanity check when you compare other time periods where there was no intervention - does your synthetic control closely match your observed outcomes? If so, you’re good to go! (View Highlight)