Dr. Mario's 2nd 🧠

đź“– Lean Analytics

6 min read

Why did I want to read it?

As I entered as Head of Data in Freepik, I wanted to refresh metrics frameworks (e.g., the ones I studied in Instituto Tramontana).

What did I get out of it?

Foreword by Eric Ries

We need “a new way to measure”:

traditional accounting metrics when applied to the uncertainties of innovation—are surprisingly dangerous. We call them vanity metrics, the numbers that make you feel good but seriously mislead. (p.xvii)

PSTD from Forecasting issues:

But accounting is at the heart of our modern management techniques. Since the days of Frederic Winslow Taylor [Taylorismo], we have assessed the skill of managers by comparing their results to the forecast. Beat the plan, get a promotion. Miss the plan, and your stock price declines. And for some kinds of products, this works just fine. Accurate forecasting requires a long and stable operating history from which to make the forecast. The longer and more stable, the more accurate. And yet who really feels like the world is getting more and more stable every day? (p. xviii)

Preface

The playbook to use this book is:

  1. Identify your business model.
  2. Identify your stage of growth.
  3. From that, choose the corresponding One Metric That Matters and draw a line in the sand.

In the Build → Measure → Learn cycle of The Lean Startup, this is the measure.

1. We are all liars

You need to lie to yourself, but not to the point where you’re jeopardizing your business. (p. 3)

Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof. (p. 4)

Management guru and author Peter Drucker famously observed, “If you can’t measure it, you can’t manage it.” (p. 4)

And in the footnote, cites Management. Tasks, responsibilities, practices.

2. How to keep score

What makes a good metric?

A good metric is:

  • Comparative
  • Understandable: “If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture.” (p. 9)
  • A ratio or a rate:
    • They are easier to act on (e.g., distance vs. speed: it gives more info about your “state”).
    • Inherently comparative.
    • Compares factors that are opposed or in tension.
    • But one must be careful not to destroy absolute figures to improve ratios! CNET deleting old content to improve avg rank.
  • Changes the way you behave: “what will you do differently based on changes in the metric?”

Metrics often come in pairs (e.g., conversion rate & time-to-purchase, viral coefficient & viral cycle time).

I was surprised by seeing “time on site” as a vanity metric:

Time on site / number of pages. These are a poor substitute for actual engagement or activity unless your business is tied to this behavior. (p. 15)

Exploratory vs. reporting metrics

An interesting view on Donald Rumsfeld “unknown unknowns” speech, applied to Data Analysis:

On an early startup, the most important thing are the unknown unknowns (in other words, exploration).

Leading vs. lagging metrics

For leading indicators to work, you need to be able to do cohort analysis and compare groups of customers over time. (p. 19)

An example of cohorts around a leading metric could be quantized number of calls to customer support in a 90-day window.

Cohort analysis

If you look at averages over time, it’s hard to know what is happening:

Cohort analysis by the month the customers arrived (for example), helps compare their experiences, and thus, whether the metrics are actually improving (here it seem that later cohorts are spending more, so it is improving):

This kind of reporting allows you to see patterns clearly against the lifecycle of a customer, rather than slicing across all customers blindly without accounting for the natural cycle a customer undergoes. (p. 26).

And of course, you can combine that with segments, see

Link to original
.

A/B testing

You can test everything about your product, but it’s best to focus on the critical steps and assumptions (…) you’ll have more things to test than traffic (p. 27).

Remember that…

It seems that in practice most tests are useless:

Link to original

And probably since you want to test many things, you may be doing multivariate analysis.

Build, measure, learn framework

3. Deciding what to do with your life

Data Team Vision and Mission:

Math is good at optimizing a known system; humans are good at finding a new one. (p. 38)

How to think like a data scientist. Common pitfalls

By Monica Rogatti, data scientist at LinkedIn(p. 39):

  • Not normalizing (e.g. not doing per-cápita comparisons).
  • Excluding outliers qualitatively (instead of only quantitatively).
  • Data vomit: “a dashboard isn’t much use if you don’t know where to look”.
  • The “Not Collected Here” syndrome (not combining your data with other sources).

[Founders] aren’t building a product. They’re building a tool to learn what product to build. (p. 41)

5. Analytics frameworks

About the sticky engine of Lean Startup:

Engagement is one of the best predictors of success (…) Stickiness isn’t only about retention, it’s also about frequency. (p. 47)

About the paid engine:

It’s usually premature to turn this engine on before you know that your product is sticky and viral. (p. 48)

The two knobs on this machine are customer lifetime value (CLV) and customer acquisition cost (CAC). Making more money from customers than you spend acquiring them is good, but the equation for success isn’t that simple. You still need to worry about cash flow and growth rate, which are driven by how long it takes a customer to pay off. One way to measure this is time to customer breakeven—that is, how much time it will take to recoup the acquisition cost of a customer. (p. 48)

6. The discipline of One Metric That Matters

Eric Ries talks about three engines that drive company growth: the sticky engine, the viral engine, and the paid engine. But he cautions that while all successful companies will ultimately use all three engines, it’s better to focus on one engine at a time. (…) Focus doesn’t mean myopia (p. 55)

Other notes

Graph View

Backlinks