Metadata
- Author: Tristan Handy
- Full Title:: Down With Experimentation Maximalism! - by Tristan Handy Down With Experimentation Maximalism!
- Category:: 🗞️Articles
- URL:: https://roundup.getdbt.com/p/down-with-experimentation-maximalism
- Read date:: 2025-03-25
Highlights
The assertion that the post from above—titled “Locally Optimal”—makes is that structured experimentation programs are good for exploiting current product innovations (finding a local optimum), but are not helpful for identifying new product innovations.
Sure, incrementally better decisions add up to a lot of value over time, but maybe we’re just stuck in a local optimum and getting many small changes right will never get us to where we want to go. A modification of the famous Henry Ford quote kind of works here: you can’t A/B test your way from selling horses to selling cars. And a corollary: if you’re testing a horse against a car, you definitely don’t need an A/B test. (View Highlight)
This is a match better expression of the idea I had regarding hypothesis testing at Freepik about whether we win the match by big swings vs small increments.
understand the context that your organization is operating in. • If the organization you’re working in is pre-PMF, you should be spending a lot of time generating hypotheses about what’s not yet working. This will likely look more like descriptive statistics, a lot of non-linear thinking, and collaborative problem-solving. You should resist the urge to rely on structured experimentation and instead be searching for insights that will unlock fundamentally new experiences for your customers. • If you’re working in a post-PMF org, expect to spend more time inside of a structured experimentation program that is designed to optimize a process that is already working. There will be more guardrails but you’ll also likely get to use more sophisticated methods. (View Highlight)
I’m not suggesting that there are zero valid use cases for experimentation in a pre-PMF organization. However, I do think that experimentation has become a shibboleth within the data community, as if a structured experimentation program is always the gold standard for how innovation is done. (View Highlight)