Systems thinking is having a moment. Netflix’s CPTO, Elizabeth Stone, recently said it’s the most important skill she looks for in the AI era. Systems thinkers are apparently “the new 10x employees.” Designers are being told it’s the one skill that will help them survive.
I’m a fan. But as I watch the hype around systems thinking reach a feverish pitch, my first reaction is: wait, wasn’t this what we were supposed to be doing already?
As far as I know, no one’s claiming they invented systems thinking recently. Elizabeth Stone points to Donella Meadows’ Thinking in Systems, which came out in 2008. What’s new is the urgency. Systems thinking has gone from a nice-to-have skill to the thing that will save your career in the AI era. Come on now, it was always mission critical. The AI hype machine needed something new to latch onto. Enter systems thinking.
When Alistair and I wrote Lean Analytics in 2013, we didn’t specifically call it systems thinking. But that’s what we were articulating: map the business, figure out where you are, identify the biggest constraint, measure it, test ways to improve it and repeat. We designed diagrams that mapped business models. We created frameworks for logically figuring out what to focus on. I guess we’re systems thinkers. Who knew? 🤷
Beyond the insane hype machine that is “everything AI,” systems thinking really does matter. AI can make individual tasks faster and output cheaper. That creates more opportunities to improve a business, and more opportunities to make one part look productive while quietly breaking another. You can generate ten reports in the time it used to take to write one. Congratulations. Who’s reading them? What decisions are they improving? How much work did you create for everyone else?
Old Ideas, New Consequences
A lot of the current “systems thinking” advice says things like, “be strategic” or “think about the big picture.” Not helpful.
The useful part is understanding how things interact: feedback loops, delays, constraints and unintended consequences. Improving one part of a system doesn’t automatically improve the whole. Sometimes it makes the whole worse.
Lean Analytics gives you a practical way to act on that understanding. The business model is the system. The stage you’re at helps identify where to focus. The One Metric That Matters (OMTM) gives you a priority. Lines in the sand tell you what result would justify a decision. Build-Measure-Learn gives you a way to test your assumptions.
Lean Analytics is systems thinking with a scorecard.
That’s the kind of discipline needed when using AI and embedding AI into your products.
What are you measuring? Why? Over what period? What would convince you that your understanding of the system is wrong? What will you do differently when you find out? Without those questions, you can spend a lot of time producing tons of output and very little time learning anything useful.
The simplest example is how people are adding AI to their work. I wrote about this in How I Use AI Without Outsourcing the Thinking. Documents are cheap to produce now, so people produce more of them, and the work of figuring out what matters shifts to whoever has to read them. At this point, everyone’s received workslop. It’s exhausting.
If you add AI to one step and only measure that step, you may miss the cost you pushed somewhere else. Measure the whole workflow: time to a usable decision or completed job, rework, quality and the effort required from everyone involved. Writing the report faster is useful if the report helps someone do something. Otherwise you’ve accelerated the production of homework for your colleagues.
The same issue shows up when AI is the product, with a few additional complications.
A Business Model Where Every Use Has a Cost
In Lean Analytics we included six business models: e-commerce, SaaS, free mobile apps, media sites, user-generated content and two-sided marketplaces. Each had a diagram showing how users move through the business and which metrics matter along the way.
Building Candor, I went back to the SaaS chapter, and it was a bit of a time capsule. We wrote that consumption pricing was “largely confined to Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) cloud computing companies today.” We also described the incremental cost of another SaaS customer as negligible. Those assumptions need revisiting for AI products.
Usage-based pricing predates AI. Variable costs do too. But if delivering your core value requires model calls, the relationship between usage, revenue, quality and cost becomes very visible. I’d pay attention to four things:
Usage can drive both revenue and cost. In a usage-based product, revenue depends on active customers, jobs per customer, units per job and price per unit. A subscription may add a fixed component, but usage still matters. More activity isn’t automatically better if you lose money delivering it.
Customers can disappear without cancelling. A pure pay-per-use customer has nothing to cancel. They just stop using (and therefore spending). Subscription customers can cancel or downgrade, but how we measure churn is changing.
Delivering trial value can incur material costs. Traditional free trials aren’t free to operate either. But in an AI product, showing someone what the product can do may require substantial computation. That cost belongs somewhere in your acquisition economics.
Successful execution and useful output are different things. Traditional software can produce disappointing results too. But AI adds another source of variability: the same workflow can run successfully and deliver outputs of very different quality. You have to evaluate the result, not just whether the job completed.
To bring this to life, I created a new business model diagram for Candor. It represents a usage-based AI product with a free trial and an optional subscription. It won’t fit every AI product, nor should it. You need to map your own business.
The beginning looks familiar. Visitors become trial customers, complete setup and reach their first result: the AI does a real job and delivers an output. But reaching first value requires another question. Was that output useful? Then comes the question that matters most at my current stage: does the customer return for another job? If they do, they become a repeat, paying customer.
In Candor, we have two pricing models: one is completely pay-as-you-go, the other has a subscription with usage fees as well. This makes the diagram a bit more complicated. Fundamentally, the key is whether or not people try Candor, see enough value, and then use it again.
Three Problems That Look Like the Same Problem
A customer runs one job and goes quiet. What happened?
Maybe the output wasn’t good enough. They tried the product, didn’t get sufficient value and decided to move on.
Maybe they don’t have another job right now. The product worked. They’ll return when the need comes up again.
Maybe they never had a recurring need. They were curious, or had a one-off task, but they aren’t the right customer.
From the outside, all three look the same: one use, followed by silence. But what you do about them should be very different. A quality problem calls for product improvements. An infrequent need might call for patience, or better pricing, along with an understanding of the customer’s timing. A customer-fit problem calls for changes to acquisition and positioning.
You can easily choose the wrong response. Improve the product for someone who has no further need, and they still won’t return. Send more reactivation emails to someone who got a lousy result, and they’ll ignore you (and potentially get quite annoyed too). Decide your positioning is broken when the customer simply needs research once a quarter, and you may abandon a useful segment too soon.
Those aren’t the only explanations. Price, setup friction, competing tools and internal approvals can all get in the way. The point is to build a set of hypotheses you can investigate, rather than assigning every quiet customer the same label.
Two feedback loops can help:
The first is sharing. Some AI products create reports, decks, code or other outputs that customers share, and those outputs can bring in new visitors. That only becomes a growth loop if people actually share the assets they create and recipients become customers. A “share” button is not a growth strategy.
The second is learning. Review outputs, identify failure patterns, turn those patterns into evals and improve the product, and better results may increase reuse. But the loop doesn’t run itself. Usage generates material you can learn from. You still have to do the learning.
Underneath all of this are the economics. Every job consumes resources, so you need to understand gross profit per job, including the retries and checks required to produce an acceptable result. Sometimes better quality requires a bigger model or additional steps. Sometimes a better prompt, better routing or less unnecessary processing improves quality and lowers costs. You need to measure both.
Dormant pay-per-use customers generate little or no usage-related cost, which makes them easy to ignore. But you paid to acquire them, and their absence may be telling you something important. Silence is cheap to host. It can be expensive to misunderstand.
What This Looks Like With Candor
Quick background: Candor is a synthetic user research tool. You describe an audience, Candor researches it and creates synthetic participants grounded in that evidence, interviews them and produces a research report.
The business is primarily usage-based, with a subscription layered on. Every new workspace gets a trial project, enough to build an audience, create synthetic users and run a real study. After that it costs $7-$10/interview depending on whether or not you’re subscribed monthly. Users are unlimited because we’re charging for research, not headcount. We also hope this encourages customers to invite more team members, who subsequently do more research. The pricing details are public.
In the “Usage-based AI product lifecycle” diagram above, an interview is a unit, a study is a job and the workspace is the customer account.
In Lean Analytics terms, we’re in the Stickiness phase. It’s live and people are using it. Now we need to answer the questions we put in the book: “Are people using the product as expected?” and “Are people getting enough value out of it?”
Growth isn’t the priority yet. As we wrote in the book, “if you can’t convince a hundred users to stick around today, you’re unlikely to convince a million to do so later.”
My One Metric That Matters (OMTM) is the percentage of workspaces that complete a trial study and then complete a second, paid study within 30 days.
People signing up and running a trial is encouraging. Someone paying to run another study is a much stronger signal that the first experience was worth something. It isn’t proof of lasting value, but it’s a meaningful next step.
The 20-30% paid conversion target is a bit high based the early benchmarks I’ve found (although there isn’t a ton of data out there about AI usage-based product conversion yet).
The “within 30 days” target is a line in the sand. I need the learning cycle to be quick. More than 30 days between studies and I’m learning too slowly.
If I can’t hit the OMTM (20-30% paid conversion in 30 days) it’s a problem that would then need to be diagnosed:
Maybe the product isn’t creating enough value?
Maybe we’re attracting the wrong customer type, which means they don’t see the value, and/or they don’t have a frequent need?
Maybe the price is wrong?
Maybe it’s a combination of these things, and/or something else I’m not thinking about?
BTW, I’m not sure if 20-30% conversion to 1 paid study is even economically viable. I know what it costs for a user to run 1 free study, and I’ll have to account for that in my overall CAC.
For now, I focus on value creation, in the Stickiness phase, measured by usage. But that alone may not be enough to build a sustainable business.
The Scorecard I Need
Reuse is my focus. That doesn’t mean I can ignore everything else. I could encourage more studies by making them cheaper, while destroying the economics. I could add reminders that increase activity without improving value. I could make the first experience look impressive while producing research that doesn’t help anyone. The scorecard needs to keep me honest.
The OMTM gives me focus. Quality and economics provide guardrails. The other evidence helps explain what I’m seeing.
Systems thinking for me, at this stage, means:
Focus specifically on what matters
But recognize that everything you do in one place likely affects everything else
To do this, I need to collect qualitative and quantitative feedback. I need to better understand customers, their needs and how Candor delivers value. It’s simple, but complex at the same time, which is what makes building products and startups so difficult.
Getting Evidence from Quiet Customers
Happy customers that use your product frequently are fairly easy to get to. Trust me, I’m doing that! I need testimonials and social proof.
You can also learn a lot from happy customers:
Why and how are they getting value? Is it what you expected and/or something different?
How can you improve the product to create even more value?
What’s similar about them that might suggest you’ve identified the right ICP (ideal client profile)? Customer segmentation is super valuable. You might be surprised.
The quiet customers are much harder to get to, but also incredibly important.
I reach out to everyone that signs up. I reach out to everyone that starts using the product. I reach out to everyone that completes their free trial. Some reply. Unfortunately quite a few don’t.
I don’t blame anyone (although it frustrates the heck out of me!) Answering a founder’s follow-up email is rarely the most important thing in someone’s day. So I’m trying to collect useful evidence inside the experience too. A few things I’m doing or planning:
Grade the first study myself. I’m already reviewing every study by hand. If an output is clearly weak and the customer doesn’t return, that supports a quality hypothesis. It doesn’t prove why they left, but it gives me somewhere specific to investigate
Ask at the moment of value. A one-click question on the report, such as “Would you use this to help make a decision?”, should get more responses than an email days later. If it does, I can see whether the answers help explain who comes back.
Ask about expected frequency. “How often do you run research?” gives me a starting point for segmentation. If someone expects to run research monthly and disappears, that’s worth investigating. If someone expects to run research a few times a year, thirty days of silence may be completely unsurprising.
Segment by customer and use case. Product managers, consultants and founders may have different needs, and concept testing and problem discovery may have different usage patterns. If one segment returns more consistently, I want to know why. It could be my beachhead.
Track return behaviour over time. Days since last study matters, but so does whether dormant customers eventually return when they have a new question, with or without a nudge from me. The second study is my current focus. The longer-term question is whether Candor earns a place in how someone does research.
None of this is particularly fancy. It’s the work Lean Analytics has always asked you to do: identify what you need to learn, choose evidence that helps you learn it and set up the business so you can get an answer. Systems thinking.
Measure What Helps You Decide
AI changes some important economics and introduces new uncertainty into product quality. It also makes it easier to optimize a task while overlooking the consequences elsewhere. That makes systems thinking useful, as it always has been. It makes measurement and judgment useful too.
If you’re adding AI to a workflow, follow the work all the way through to a usable outcome. If you’re building a usage-based AI product, understand how quality, frequency, pricing and cost interact.
For Candor, the question is whether the customer got enough useful value to pay for another study when they have another research job to do. The scorecard helps me investigate that question. It won’t answer it for me. That’s the work: form a hypothesis, collect evidence, make a decision and be willing to change your mind.
If you map your own AI product this way, I’d love to see it. And if you think I’ve got Model Seven wrong, tell me. It’s version one. I expect to learn something.