Test First, Build Later: A Guide to Validating Your Ideas With Stimulus & Prototypes

Use "throwaway" concepts, visual aids and quick experiments to learn fast, before you build and launch your product. (#20)

I’m not sure what’s happening here, but it’s definitely scrappy!

I love building things. Working with designers, developers and product managers to take an idea and turn it into reality never ceases to amaze me. And it never gets old or boring (even building incremental feature improvements is cool.)

But building the wrong thing is brutal.

You still get that quick high from shipping, but when you realize you’ve missed the mark and too few people actually care, it’s a punch in the gut.

Have you ever built something, launched it and then…nothing?

I always picture tumbleweeds going by a deserted town in an old Western movie. Or in a Simpsons episode. God, I hate tumbleweeds.

I remember launching Standout Jobs in 2008, and after the initial rush, which included a fair bit of positive reinforcement from the recruitment industry congratulating us for launching, it was pretty f-ing quiet. We felt like heroes for all of ten seconds, and after that it was a mad and very uncomfortable scramble to figure out where we went wrong. We were on-trend, developing a reputation in the market for being innovative, and had an amazing team. But too few customers signed up. And those that did, struggled to use our product.

Basically: we shit the bed. 💩 🛏

But this was after a considerable amount of effort, and it was difficult to do anything other than keep adding features to the core product in the hopes that we’d land on something people really wanted and could use.

Note: I should point out that sometimes people want something, or say they want something, but then when you give it to them, they don’t use it. There are a few potential reasons for this. People tend to be pretty lousy at articulating solutions. They are much better at sharing their problems (generally in my experience, people like talking about their problems.)

For the most part we got a lot of tumbleweeds.

The industry was still hailing us as an innovative disruptor, and superficially a lot of things lined up, but we couldn’t get enough traction to scale. Hype is hype. That’s all it is. Once you buy into it, you run the risk of believing it so much that you forget the fundamentals.

Standout Jobs wasn’t my only tumbleweed experience. 😔

And I’ve vowed since then to avoid that experience (and the subsequent depressing feelings) by any means necessary.

This is what led me to dive head first into the world of Lean Startup. While I don’t believe in being a process zealot, I’ve come to appreciate how important it is to:

1. Break big things down into smaller things

2. Focus on the riskiest assumptions first and how to validate them

3. Run rapid experiments in an attempt to learn faster

4. Be more intellectually honest with myself and my team

5. Minimize how much time and money we waste

6. Not start a company on a level playing field

In a world where it’s insanely easy to build new things, the notion of actual differentiation that matters to users/customers is going out the window.

How many new Generative AI tools have launched in the last month alone? 500? 5,000? 50,000? It’s impossible to tell. They’re all cool experiments, but very few will survive and become actual businesses.

The counter to a sea of sameness is focus & learning.

One key aspect of focus is going after a specific vertical deeply, as opposed to building horizontal platforms out of the gate. If you’re trying to solve for everyone, you’re solving for no one.

And learning means being faster & better at deriving actual insights from the market. You could argue that shipping a product super quickly and then learning is the way to go, but in my experience as soon as you launch, you are more attached to the product than before, and that attachment grows into a near-overwhelming force, making it harder to be intellectually honest, learn fast and pivot meaningfully. Launched products have a way of calcifying people into blindly believing they’re on the right path.

Y Combinator is the ultimate accelerator program. I know lots of people that have gone through it and had an incredible experience. It changed their lives. And recently their Pocket Guide of Essential YC Advice was shared on Twitter. It’s a fantastic resource.

So what should you do?

Don’t build a product.

Instead, invest your time in validating your riskiest assumptions through stimuli & prototypes. These are artifacts you create for the sole purpose of learning something, with no pretense that you’re keeping the stimuli or prototypes alive. Once you’ve learned from them, you throw them away.

Let’s quickly define stimulus & prototypes

There’s definitely overlap between the two, but please don’t get caught up in the definitions. At the end of the day it’s about picking the right tool for the right job (more on that later.)

First, what’s a stimulus and prototype?

• A stimulus: This is the scrappiest thing you can create or use. It rarely resembles your actual solution (you may not even have a solution in mind.) Think of a stimulus as a visual aid to help you learn from users. Occasionally, we call a stimulus a “concept test” as well. Examples: existing products, mood boards, concept statement, paper prototype, sales sheet, journey map.

• A prototype: Typically slightly higher fidelity than a stimulus (but again, there’s overlap), where the user is going to get a better feel for the solution (and likely be able to interact in some way.) The usability of a prototype makes things feel more real. Examples: landing page, ads, clickable prototype.

NOTE: Neither of these constitute a Minimum Viable Product. I will be writing something on MVPs in the future, because I believe the definition has gotten so confused that no one really understands it any more.

OK, I’m ready to start using stimuli & prototypes to test things before building an actual product. What’s next?

1. First, you need to define your riskiest assumptions

• To learn how, read this: How to Apply the Scientific Method to Startups Without Being a Zealot

• Then use an assumption template (Google Sheet) to list out and rank your riskiest assumptions. (Note: I’ll share a Notion version below if you’d prefer to use that.)

Being intellectually honest and rigorous about your riskiest assumptions is critical. If you chase small issues that don’t really matter, you might feel good about the early results you get, but ultimately you’ll go down the wrong path.

2. Now figure out how to test those riskiest assumptions using stimuli & prototypes

You should use stimulus immediately — as soon as you start validating a problem — especially when conducting user interviews. I like kicking off a handful of interviews without any stimulus, and then putting together simple visual aids to help me in the next batch of interviews. I’ve often found that having a visual aid is very effective at teasing out quality insights from users versus simply speaking with them. There’s something about engaging people with something “physical” (it could be a digital stimulus) that disarms them when they’re talking. It also helps bridge the gap from “say vs do” (i.e. what people tell you they’re interested in or will do versus what they actually do.)

There are many tools in the proverbial toolbox. It’s not always easy to know which one to use. A few things to remember:

1. You will almost certainly run experiments that don’t work. As long as they don’t take a long time or cost a ton, it’s OK. You’ll still learn something.

2. You will most likely need to run multiple experiments per assumption. You can do this in parallel or asynchronously. I tend to lean towards volume > perfection, because I’m trying to gather as much learning as possible, and then synthesize it after.

3. You will eventually string several concept tests & prototypes together. For example, you may have Facebook ads going to a landing page converting into a survey. These start to become more complex tests.

4. Try and isolate each assumption and test it independently first. This is hard to do, but give it a try. For example, if you’re at the stage where you’ve already envisioned the MVP, you can break it up into its individual features/value propositions and experiment against each one without having to build the MVP first.

3. Don’t get caught in analysis paralysis

Taken to an extreme, the experimental methodology described above may lead you to never building and shipping anything. While it can be extremely helpful to try and break everything into small pieces and test each individual thing (or assumption), it’s never that simple or clean.

At some point you have to go with your gut and build something. While applying some scientific rigour to this process is important, it’s easy to overdo it.

If you decide it’s time to build, then do your best to keep the build simple. This is where we get into the discussion and debate about MVPs. What is an MVP? Do MVPs work? Have we lost the plot on MVPs? Sigh. Again, I’ll be tackling this in the near future.

Before you do build an MVP, I hope you can find ways to learn quickly through rapid experimentation leveraging stimuli or prototypes. I can almost guarantee—even if you’re not great at this stuff—you’re going to learn something. And, the more you practice, the better you’ll get. So the first handful of stimuli or prototypes you create may not work, but they’ll still be faster and less expensive to execute than building an actual product.

In summary:

1. Test first

2. Build later

3. But don’t test too long or try and figure everything out before you build (there’s a limit to what you can learn without an actual product)

4. All things being equal do something; while you may waste some time doing this, I’m fairly certain you’ll waste less time doing something than doing nothing. 😃

Here’s the Focused Chaos Experiment Toolkit!

I know it’s not always easy to know what tool to use to run experiments. When is the right time to build a landing page? What about a clickable prototype? Is using a simple stimulus too cheesy or lame? Shouldn’t I just code the damn thing already and see how people react?

In an attempt to help out, I’ve built an Experiment Toolkit, which is a list of stimuli and prototypes (I’ll add to it over time) you can use to run experiments. It includes brief descriptions, categories, links to additional resources and more.

The Experiment Toolkit also includes an Assumption Tracker template.

Unfortunately I can’t embed the toolkit into Substack, so I’ve provided a snapshot image below. I know, it doesn’t look that cool, but hopefully you find the resource helpful!

Here’s a direct link to the Experiment Toolkit: https://experiment.focusedchaos.co

If you have any ideas for stimuli, prototypes or experiments that I should add to it please let me know.

---