Build, Measure, Learn: The Expanded Edition
Introducing the Lean Analytics Cycle, a way to experiment successfully. (#78)
Startups always have more problems than solutions. In fact, every time you solve a problem, it leads to another one. Everyone’s job in a startup is nonstop (often high risk) problem solving.
On top of that, there are always more opportunities than resources. This is true across product, marketing, sales, etc. Do you build A or B? Do you spend money on X or Y? You’ve probably heard the phrase, “Constraints breed creativity.” Sure, but that doesn’t make the endless decision making any easier.
Building a startup is an exercise in problem solving and decision making.
That’s true for every single person that works at a startup. And the magnitude of each problem solved and each decision made is big, because there’s nowhere to hide. No pressure, right? 😀
Build → Measure → Learn
Most people are familiar with Lean Startup and one of its core principles: Build → Measure → Learn.
First you figure out what to do, ideally something small and quick. Then you build that thing, measure the results and learn. The more you iterate through the process, the higher likelihood you figure things out. Cycle time is key. If experiments (and pretty much everything is an experiment) take too long, your speed to learning is slow, which is bad.
Build → Measure → Learn is simple and elegant. It encourages an experimental approach and the right mindset for validating each piece of the puzzle independently before making huge leaps forward.
Unfortunately, like many frameworks, it’s oversimplified. It’s only one component of Lean Startup, but taken on its own you’re left with lots of questions:
How do we decide which idea(s) to pursue?
How do we know if an experiment is the right one?
How do we actually measure results?
How do we codify our learning?
Etc.
Inspired by Lean Startup, Alistair Croll and I expanded on build → measure → learn and created the Lean Analytics Cycle.
The Lean Analytics Cycle
Imagine double-clicking into every aspect of build → measure → learn and it might look like this:
This may seem a lot more complicated, but it’s not.
1. Pick Your One Metric That Matters and Draw a Line
The One Metric That Matters is the key metric you’re focused on today. It will change in the future as your startup evolves. But if you can only focus on one thing right now, what should it be? Focus wins. Chasing shiny objects or doing too much at once likely leads to failure (unless you get really lucky). 🍀
To pick the OMTM, decide on your biggest problem. What’s the #1 thing holding your business back? A lot of founders focus on the wrong thing—they go after small problems or opportunities that won’t solve the fundamental challenges. A great example: Chasing buzz when your existing users don’t use your product enough. Sure, buzz feels great, and leads to an injection of new users—but if the product isn’t valuable enough for those users (like it wasn’t for the existing / previous ones), churn will be too high and you’ll fail.
In my experience most founders know what their biggest problem is, they just don’t want to admit it, or aren’t sure how to fix it. So they phutz around the edges hoping for a miracle. But they know.
If you identify the key problem to focus on, you can figure out the related metric. Try mapping your business as a system, it’ll help:
Once you’ve got your OMTM, draw a line in the sand for your goal. Let’s say churn is 10% per month, which is too high to justify doing any real customer acquisition. You do some research and decide that getting to 5% per month is a reasonable target. That’s not to say 5% churn/month is fantastic, but from what you can tell that’s good enough to warrant opening up the top of the funnel and focusing on acquisition.
2. Find a Potential Improvement and Write a Hypothesis
Now figure out how to lower churn. This is a great opportunity for an ideation session with the team. It’ll get everyone focused on the same thing.
You can run these sessions (virtually or in-person) in a variety of ways, but here are a few suggestions:
Have all teams/departments participate (if you’re divided that way). Churn is everyone’s problem from sales to customer success to engineering and so on. Aim for 5-8 people per group (any more and it’s chaotic).
Give everyone 5-10 minutes to write as many ideas as possible. There are no bad ideas. Everyone does this independently and quietly.
Then go around the room, starting with one person, and have them share their ideas. Cluster similar ideas from others (on a virtual or physical whiteboard/wall). Allow for a bit of context sharing and discussion per idea. This should take ~30-45 minutes.
There may be so many ideas that you need to group them into themes. Take 10-15 minutes to do so.
Now give everyone 3-5 votes to pick the ideas they believe will have the most impact. You may get some groupthink, but that’s OK. Encourage people to vote without paying attention to what others are voting on (i.e. their bosses).
Rank the ideas with the most votes and refine them. Discuss the top 3-5 ideas further, or do that later. The goal of this exercise is to get all the ideas on the table and give everyone a say.
Another way to look for good ideas is in the data you have. Look for patterns that might suggest experiments you can run.
What are your best users doing compared to those that churn out?
What commonalities exist amongst your best users? Maybe they’re all within a specific industry? Maybe they all work at similarly sized companies?
Brainstorming exercises always result in too many ideas. Remember: You have to focus and can’t do everything at once. Even the top 3-5 ideas may be too many. Ideally you run them as individual experiments to isolate what really works.
Go through a rapid prioritization exercise to assess value versus effort. Define how much each idea will improve things. In our case, which idea will reduce churn the most (while also requiring the least effort)?
Write assumptions for the top ideas. This is an important part of the process because it forces you to be rigorous with experimentation. Here’s a full post on how to use assumptions properly:
Use a simple assumption / hypothesis tracker ⬅️ that’s a free tool you can use. Write your assumptions down. Make sure everyone has access to them. Alignment with the team is key.
3. Design a Test or Make Changes in Production
Now put your ideas to the test. You may do something more sophisticated like run an A/B test (i.e. release a new feature to a portion of your user base), provide early access, conduct a fake door test or something else. Here’s an Experiment Toolkit for inspiration.
There are lots of ways to test things without releasing new features. Maybe you can remove a feature. Or change a process. In the case of churn, if you realize that your best customers have a specific commonality (same industry, geography, etc.) then target similar customers. Or shift your value proposition.
It’s easy to overthink things. After all the ideation, brainstorming and prioritization, don’t get caught in analysis paralysis. Pick an experiment and push it live to production for everyone. This is one of my favourite emojis in Highline Beta’s Slack (feel free to steal it!):
Generally, aim for experiments to take no longer than 2 weeks to build. Two weeks is a good sprint length. Anything longer and you want the team to ask itself whether things can be done more simply or differently. Sometimes things take longer, but recognize that your cycle time is also increasing, which reduces the volume of learning, and that’s a risk.
💥 Time for an experiment of my own…
I signed up for Intro, where you can book time with experts on a variety of topics. Need help with product, raising capital or startups in general? I may be able to help.
➡️ Book time with me here: https://intro.co/BenjaminYoskovitz
4. Measure the Results
With your experiment launched, it’s time to measure results. In reality, you’re probably running multiple simultaneous experiments because otherwise it’ll feel like things are going too slow. This makes it tough to determine what’s really working (or not), but startups aren’t in a controlled lab. Life is messy. Accept it and do your best.
The big question is this: Did we move the needle?
Churn was 10%/month and our target is 5%. We ran an experiment. What does churn look like now?
Note: Churn is a lagging indicator, which means it takes time to measure. This makes it tricky to experiment on. On the other hand, customer complaints is a leading indicator—i.e. if they’re going up, there’s a good chance churn goes up as well. Churn may be the One Metric That Matters, but perhaps you can measure it initially through customer complaints. If you went through the brainstorming exercise with the full team, I bet there were some ideas around reducing customer complaints.
Let’s say your experiment failed. Churn is still 10%. You could give up, although that seems like a radical choice. Maybe a pivot is in order.
A pivot is a shift in one aspect of your startup’s focus, based on validated learning.
Churn stayed at 10%, but then you realize it actually dropped for a small segment of your customers. It didn’t show in the average, but through proper data analysis it’s clear. That’s an important learning. It might suggest a pivot towards a certain type of customer. While this could feel like a scary move—you’re abandoning a “bigger market” to focus on a narrower one—in the end it might work.
Q: Is it better to have a bunch of happy customers in a smaller market or a bunch of unhappy customers in a bigger market? A: The former.
What if churn went to 5%? 🎉 🥳
Celebrate. Hard. Then get back to work.
You know 5% isn’t the end goal, but it’s good enough to justify moving onto something else. What do you do after you’ve fixed the leaky bucket? Pour more into it!
Now look higher up the funnel and figure out how to increase the number of leads, or increase conversion through the funnel (to activation and ultimately, retention). You’ll decide on what to focus on based on the data. Or you’ll go with your gut. Or both. Which part of the funnel is the most broken? Start there.
Let’s be honest: the odds of going from 10% to 5% churn after 1 experiment are very small. More than likely you’ll try again. 💪
There’s a good chance your experiment works, but not enough. Maybe churn drops from 10% to 7%. Back to the drawing board. You take the learnings from the experiment (or experiments) you ran, review the prioritized list from before and move forward with the next tests. Maybe you go through another brainstorming and/or reprioritization exercise based on what you’ve learned. Or you simply go to the next idea on the list.
The more experiments you run, the higher likelihood of diminishing returns. It’s possible the Nth experiment is the winner, but more than likely churn drops less and less after each test. Maybe you get churn from 10% to 5%, but keep an eye out on effort. If you’re at 5.5%, is it worth the effort to get another 0.5% reduction? What if it takes months to accomplish? What if you invest tons of time and money and it never happens?
At some point you draw a new line and move on. Perhaps 5.5% churn is fine—you don’t have any great ideas for how to lower it and you can’t ignore the rest of the customer journey / funnel forever. Some might call this cheating. I think it’s being pragmatic. (For an example of this read the story of Highscore House.)
As I said in the beginning, once you’ve solved one problem, it simply leads to the next one. In Lean Analytics we said “metrics are like squeeze toys” because when you apply pressure in one spot (fixing something) it logically reveals where to focus next.
Happy problem solving, decision making and squeezing!
Great article Ben!! Very very Practical. Is the way I see things when trying to execute in the chaos. In my philosophy of work, the iteration must be a week long. Anything that lasts more than a week to build, is because:
a) the task is not well defined in scope hiding more tasks in it (kind of a Trojan Horse)
b) A bad task with too many blind spots or dependencies that the team should cover up without strategy .
c) any piece of work of any kind isn't long to execute in more time than a week. If it does, get back to a)
Really great and insightful article.