We’re either swimming in a lot of data or opinions. It’s tough to find the happy medium.
On one hand you have those that believe every decision should be data-driven. On the other, you have HiPPOs (otherwise known as Highest Paid Person’s Opinions.) 🦛 📢
Independently, both are frequently wrong.
The key is finding a balance and ways to combine data + opinions together.
1. Data as an Input
Data is an important input in the decision-making process.
As soon as you have a product in-market, you can collect meaningful quantitative data that can impact your product roadmap and much more.
Let’s say you take an iterative approach to product development (applying basic Lean principles of Build → Measure → Learn.) This lives at the core of how you work.
“Ideas” have to come from somewhere. There are a host of inputs (more on that later) but data is one that many product managers rely on.
Product managers know they need to instrument their applications (tracking events) to collect quantitative data. This helps you learn how people use your product, and who is the most active.
5 Ways to Use Data Properly
Map your business as a system: I like to visualize businesses as systems diagrams, mapping each of the steps & interactions with users and customers. The systems diagram allows me to identify trouble areas, priorities and figure out what metrics to use.
Instrument as you build: Don’t build the product and then instrument it later. If you do them at the same time you’ll be more purposeful and thoughtful about it. At minimum, identify the key user flows and make sure those are being measured.
Write PRDs with data in mind: When crafting your Product Requirements Docs (PRDs) or other specs (including epics and stories), include the data that you used to make decisions. And include the metrics you’ll use to measure impact.
Find benchmarks to help you: Good metrics are comparative. But what do you compare your data to? Seek out benchmarks online. For example: B2B SaaS benchmarks. Over time you can use your own data as a benchmark too.
Get familiar with cohort analysis:Cohort analysis is how you compare different groups of people over time. It’s most commonly used for determining churn/retention, but it can be used to measure pretty much anything. If you only use averages across all users you’re not going to get as fine an understanding of what’s going on.
You can make a lot of decisions off quantitative data, but there’s a big risk.
Quantitative data doesn’t give you enough insight into what’s really going on. It doesn’t answer the question, “Why?”
A key point to remember:
Warning: Data alone often optimizes for incremental improvements
People who rely exclusively (or almost exclusively) on quantitative data usually end up obsessing over incremental changes. If you have millions of users, small incremental changes can make a big difference, but it’s rare. Data obsession leads to the belief that small tweaks based off the data can generate enormous results. Maybe…but you also get death by a thousand paper cuts…
Without marrying qualitative data and quantitative data you’re not seeing the whole picture.
If only it were that simple. 😅
Within “qualitative data” there are a host of actors. The HiPPO emerges, along with other departments within your company, industry trends, competition, partners, etc. All of this input is overwhelming, but necessary.
While this diagram is reasonably neat, most of the time product managers face the tsunami of input in a much more chaotic way. It’s tough to navigate.
But here’s the point: You’ll get a lot of qualitative information (including random opinions) from a host of sources. You’ll need to manage all of that plus the data you’re collecting to make good decisions.
This is where data becomes a filter (along with being an input.)
2. Data as a Filter
filtering the noise
No one likes being constantly peppered with information. In my experience it’s inevitable for product managers to face the barrage.
Some product managers rely too much on data as an input, in an attempt to quiet all other inputs. This is very tough. Everyone that’s less data oriented will push against this, and data (as we’ve mentioned above) has holes. By itself, quantitative data doesn’t give you enough insights into what’s going on.
I like using data as a filter. It’s the “great balancer” between all the disparate voices.
Data takes a lot of the opinion out of things. And that’s what we need if we’re going to make constant decisions, quickly, and be able to iterate & learn through experimentation.
I don’t believe you should completely eliminate your guts & instincts. On the contrary. Product managers are in a position to connect more dots than anyone because they’re at the epicentre of all the action. They’re collecting and reviewing all the feedback, which should make them better at seeing patterns and making decisions, even if the data evidence isn’t quite there. Pattern recognition and dot connecting is how you hone your guts & instincts (it’s not just random thoughts in the shower.)
If you don’t have the data available as a filter, you can either:
Move ahead with a decision any way; or,
Collect the data first, and then review & compare it with the qualitative input
Pick a path based on the scope of work ahead of you. If it’s a quick experiment or something you can implement quickly (1-2 weeks) it might be best to move ahead, even if you don’t have great quantitative data. If you’re talking about a bigger feature or implementation, get the data you need to make (or not make) the case.
Using data as a filter helps you instrument your product better. It helps the whole team improve the use of data in strategy sessions, product planning, scoping work and testing. “Do we have data that suggests this is the right way to go?” is a reasonable question that most people should ask.
Data as a filter will help you make better, less opinionated decisions (and mitigate the HiPPOs and other voices in the room), but only if you can use data to communicate effectively.
Everyone in your organization needs to “speak data.”
If your data is too complex or inaccessible it becomes “dark magic that some people in the back corner work on.”
In that case, no one will trust your data, and all the effort you went through to collect it and set up your product teams to be hypothesis & experiment driven will be for naught.
People don’t trust things they don’t understand. As startups scale, silos between teams & departments emerge. Those silos lead to trust breakdowns. Data helps, if it’s simple to understand and accessible.
Going through the exercise of mapping your business as a systems diagram, helps with overall alignment. Then you attach specific metrics to each part of the business, and get alignment on that.
Sidebar: How to map your business
Mapping your business may look complex but it’s quite straightforward. Here’s an example of a freemium SaaS business. You can imagine a customer going through this whole journey, with an associated metric to track at each step. As the business pivots and scales, the map changes.
I know I’m oversimplifying. You may have complex data analysis and queries going on, with data strewn all over, etc. Nothing is ever as easy as the frameworks or visuals suggest. But if you’re in product management or involved in product in any way, you need to find ways to simplify data and use it to create common understanding amongst everyone in your company.
Data can also help align and motivate employees. It gives them collective goals and clarity on what you’re trying to achieve. OKRs are one way organizations try and use data to clarify what everyone should be focused on. I like to use the concept of stacked One Metrics That Matter, where each project and department/group have a key metric they’re looking to achieve, which bubbles up to a key metric for the whole organization. You can read more about the approach below:
There are a variety of frameworks you can explore to figure out the right things to track within your company. Jason Cohen recently published an interesting approach, “Selecting the right product metrics.”
The key is this:
Don’t overcomplicate the data
Make sure you use it as an input and a filter for decision-making
Balance qualitative and quantitative data, and make sure both are represented throughout your product management processes
Data can break down barriers and reduce the chaos (and the HiPPOs dominating everything) if everyone understands & trusts the data
Communicate often in terms of how the business operates, where it’s main problem areas are, and how you’re measuring progress so everyone is on the same page
Don’t miss out on more free content! Subscribe below. Thank you.
