Vanity Metrics: The Numbers We Hate to Love
Discover more from Focused Chaos
Vanity Metrics: The Numbers We Hate to Love
If vanity metrics are bad, what makes a good metric? How do I know I'm tracking the right things? (#12)
Ah, Narcissus. The poor bastard fell in love with his own reflection and died staring at himself. What a way to go…
There’s a bit of Narcissus in all of us (some more than others!) And when talking about startups, narcissism rears it’s ugly head in the form of vanity metrics (and obscenely fancy offices with on-tap kombucha and massage chairs, but that’s a story for another time.)
Before continuing, I’m going to make a counter-intuitive (and perhaps blasphemous) statement:
→ Vanity metrics are useful (sometimes.)
Don’t leave just yet, hear me out… 🙏
What are Vanity Metrics?
Vanity metrics are numbers that make you look good, but have no material impact on your ability to make good decisions and improve your business. You can’t use them to affect strategy or learn anything. They’re the numbers that always go “up and to the right”—beautiful on a graph, but otherwise mostly useless.
Ultimately, vanity metrics cannot change your behaviour. This leads to the “Golden Rule of Metrics” (coined in Lean Analytics):
Here are some examples of vanity metrics:
# of users
# of social media followers
# of likes on a Twitter post (or any social media)
app downloads
page views
newsletter subscribers (But I still want more! Please subscribe below👇 😍)
These are all vanity metrics because they’re not a measure of actual engagement or value creation. You can acquire more users, social media followers or app downloads, but if those people don’t do something after, who cares?
I have ~16,700+ Twitter followers, but many of them are legacy followers from years ago. As a result, my Twitter engagement isn’t super strong. Those 16.7k followers = total vanity (but I want more! 🐷)
Arvid Kahl does a good job of explaining vanity metrics in the context of creators. In this space, vanity metrics run rampant, because there’s an assumption that you need “massive” followings to create value.
You’re still going to measure vanity metrics, often at the top of a funnel, but optimizing for them is bad news.
For example, website traffic, is a vanity metric. Of course, you need traffic to your website in order to convert people into users/customers, but growing traffic alone won’t help you win.
So what makes a good metric?
A good metric has 4 key qualities:
Understandable: The more data we can track, the more complicated things have become. This isn’t helpful if you want to use data in your startup as a common language. A good metric is one that’s easy for everyone to understand and track.
Comparative: A good metric allows us to compare things. Comparative numbers—typically over periods of time—allow us to see more accurate trends. This is often what we think of when we talk about cohort analysis. For example: Active Users vs. Active Users/Month. If I tell you I have 10,000 active users it’s difficult to know if that’s good or bad. If I tell you that last month I had 1,000 active users, that’s a 10x increase, and that looks pretty good!
Ratio / Rate: Take a comparative number and turn it into a ratio or rate and it becomes even more valuable. Using my example above, instead of Users/Month, I should track % Monthly Active Users. So last month I had 1,000 active users out of 2,000 that joined my platform, which is 50% monthly active users. This month I have 10,000 active users, but out of 1,000,000 signups, which is 1% monthly active users. So I have a lot more active users (which is good) but something went horribly wrong with conversion.
Behaviour changing: We already covered this above, but as a reminder: a good metric is one that you use to make decisions. Imagine looking at a metric and thinking to yourself, “If this goes up, stays the same, or goes down, I don’t know what I’d do differently.” ← stop focusing on that metric.
The best (worst) vanity metric of all time
Many years ago I was doing a presentation about how to combine product management, design and Lean Analytics. As I was speaking about vanity metrics, one of the attendees raised her hand and said, “I may have the all time ‘best’ vanity metric.”
She worked at a large bank where they were focused on building a mobile app. The project leader felt like they were behind the competition and panic was setting in. They weren’t doing any real user research, and instead were in “copy mode.”
As a result, they were building as much as they possibly could. They were going to compete on the number of features, and the project leader celebrated every new feature that was stuffed into the product.
As ridiculous as this is, it happens a lot. And not just in large companies. Startups often put a ton of emphasis on building & launching features, because they believe it’ll automatically create more value for users/customers. It rarely works.
This is why you hear the argument of outcomes over outputs. The goal matters more than the amount of work or the speed of the work to get there. Just “getting something done” is not super valuable, unless it moves the needle. I tend to agree with this—ultimately, if you can’t deliver value to users/customers, you’re wasting your time. And I’ve written about this before.
But I wouldn’t say “outputs” are vanity, because in a lot of organizations (big and small) they struggle to even get things done. When that happens, the simple act of doing the work and shipping something new should be considered a victory, because without outputs there’s zero chance of outcomes.
To be 100% clear though: # of features = a ridiculous vanity metric! 😆
Is the amount of capital you’ve raised for your startup a vanity metric?
Yes.
We’ve all seen the announcements of startups raising capital, especially when it’s a big amount. “Acme just raised $XM!” Woohoo! Everyone celebrates! 🥳️ A couple years later, they’re dead. 💀
This happens over and over and over and over again.
Fundraising announcements may help attract customers, employees, partners, etc. but they’re still largely exercises in vanity. And the amount raised? Vanity metric.
Here’s an excerpt from a recent Forbes article on Cometeer, a venture-backed darling that was going to completely disrupt the coffee industry:
To the outside world, Cometeer was buzzing. Its coffee—roasted by elite partner brands like Counter Culture and Joe Coffee, flash-frozen and shipped in recyclable aluminum pods that needed only to be dropped into hot water to serve—was beloved by tech influencers and venture capitalists alike.
When Cometeer announced $35 million in funding from name-brand investors in October 2021, taking its total raised to $100 million, the startup became the highest-funded coffee startup ever. “We’re hiring folks from Apple, Tesla, Palantir and Wayfair,” founder Matthew Roberts told Forbes then. “We are talking about real tech companies, real tech employees coming into the coffee industry because they see the opportunity to change an industry that’s really been stuck in its old ways.”
Are you as hyped as I am?!?! 🤑
(btw, the title of the Forbes article is: After Layoffs And A CEO Change, Cometeer’s Frozen Coffee Pod Business Is In Hot Water — so ya, you probably know where this is headed.)
Granted, the amount you raise does change your behaviour.
If you raise 💰💰💰💰💰💰💰💰💰 you spend 💰💰💰💰💰💰💰💰💰
If you raise less, you tighten your belt
But the amount you raise doesn’t drive learning. It might give you a chance to run more experiments and learn from those, but usually it leads to crazy spending on stuff you don’t really need. So more vanity.
Should I ever use vanity metrics?
Yes. 😱
As I mentioned above, vanity metrics are occasionally useful.
1. Vanity metrics are useful for the press & media
It’s rare that you ever see any metrics that matter in a press release or a news story about a startup. Most of the time these are vanity metrics. They’re the types of metrics that make everything seem rosy and wonderful, regardless of what’s happening under the hood.
It makes sense to obfuscate important metrics from the press or media; those metrics can be used to understand core strategy (which you don’t necessarily want to share), or the underlying metrics aren’t really great (and you don’t want anyone knowing.)
So by all means, share your vanity metrics publicly and let everyone gasp in awe at your awesomeness. (Then get back to work.)
2. Vanity metrics find their way into investor updates (but be careful!)
Investors aren’t immune to a little vanity. 🤣
I get a decent number of investor updates from startups that include vanity metrics. Most of them admit when they share vanity metrics, but they still do it. It’s not great, but it’s not terrible—sometimes we all need to believe that things are headed “up and to the right” a little more than is truly the case.
Sidebar: Sometimes “good metrics” are total nonsense
With early stage startups, the numbers are almost always small—these startups don’t have a lot of traction. So showing something like 20 users isn’t compelling. Instead, these startups use “good metrics” (namely comparative ratios/rates) to make numbers seem more exciting.
“We have 100 users, and last month we had 50” — 😴
“We have 100% MoM growth” — 🎉
Come on now…sneaky founders; we (VCs) can do math (most of us.)
3. Vanity metrics are often needed to calculate other important metrics
A good metric is comparative, and ideally a ratio or rate. For example: conversion metrics.
A conversion metric—say of visitors to buyers on your e-commerce website—is a useful metric, and it uses a vanity metric (site visitors) in its calculation.
So there are some vanity metrics that can be used to calculate other valuable metrics.
App downloads is one of the first metrics at the top of the funnel for mobile apps. Without it, you can’t calculate % signups (you’d have raw # of signups, but no clarity on what % of users sign up after downloading.) If all you have is signups, you can still calculate important metrics down the funnel (the next one being some form of “activation”) but everything starts with downloads. So measuring downloads is worthwhile—optimizing for it is a different story.
Bonus #1: Is brand awareness a vanity metric?
Several weeks ago, I asked people to share their “favourite” vanity metrics. Jacquelyn Corbett, Founder at JC+CO, which offers Fractional CMO and implementation services and has built, launched and scaled 60+ omnichannel consumer brands in the last 5 years (including Sheertex, Flow Hydration, Crafty Ramen & Rebelstork) replied:
“I once worked with a team that was constantly reporting that sentiment for the brand was on a major incline. Except it weirdly didn't ever correlate with sales, social growth, partnerships, retail or email activity, or literally anything but their own reporting.”
She and I started talking about how companies measure “brand awareness” and how much of that is vanity or valuable. It resulted in this Q&A:
Ben: How do you measure brand awareness properly so it's not vanity?
Jacquelyn: It's really challenging to measure brand awareness for early stage brands, so we cobble together metrics as we're able to. I like to think of awareness as a spectrum, from vaguely aware (i.e. reached) to actively aware.
Most brands do measure some form of brand awareness, but it’s often vanity. You can buy followers, likes, comments. Everything is for sale, which is why you often see insanely low conversion rates. Brand teams or agencies are purchasing these social media bumps on the front end—not necessarily in bad faith—but to create more immediate brand credibility when a prospect lands on a social feed. People trust brands with 10k followers more than a brand with 25 followers, whether or not those 10k people are real.
Ultimately you have to properly measure brand awareness first, and then focus on measuring various micro-conversions. If you’re building brand awareness but focused on measuring conversion-related metrics you’ll be disappointed.
A few of the techniques we combine to recognize whether brand awareness is shifting include:
Direct channel traffic: When people are populating your URL directly into the browser, they definitely know you exist. If you monitor this channel's growth, you can use it as a reference point for brand awareness.
Search volume: When people search your brand name or recent campaigns directly, they know you exist. Keep an eye on Google Search Console and monitor trends.
Advertising engagement: Reach is a rough reference point and heavily impacted by spend without a real sense of whether or not the reached recipient had any recollection of the brand. Video view-throughs (targeting minimum ~25% video completion) and clicks to site give us a more reasonable reference point to awareness.
Social engagement: It's really easy to “fake up” total followers, and while reach is helpful, it's a super tricky one to treat as an awareness metric (though useful for other reasons.) The ratio of accounts engaged/accounts reached is a potentially more interesting metric for understanding awareness in social, because it gives you a sense of how many people actually saw the brand content you produced and had a response to it. Sentiment also matters on the social side—if engagement is up because people are complaining, that isn't an improvement in brand awareness. When your entire team starts commenting constantly on your posts, that creates a false narrative as well. With social media it's super important to monitor and understand the results being reported.
Brand owners need to keep a close eye on whether metrics like increased social activity convert over to actual micro-conversions. For example, we'll do things like run regular content in social channels to get followers to join email lists or engage with landing pages. If we can't generate these micro-conversion, I assume much of what's happening in-channel is vanity-driven and I wouldn't rely on it to be part of a proper conversion funnel.
Another thing I see is more traditional campaigns generating tons of views on sites like YouTube, followed by minimal traffic to site. In these cases, you can make a case for increased awareness, but if, within 30 days, you're seeing negligible movement in site metrics like direct traffic or organic search, then it's unlikely that it had a lasting impact on awareness.
Ben: How do you balance building brand and optimizing conversion later in the funnel?
Jacquelyn: In my experience, it's as critical to build awareness as it is to drive conversion—the funnel is wider at the top and if you're not filling it, you wind up with a weak lower funnel and a bloated CAC.
We often see digital marketers looking at the funnel on an audience basis; i.e. top of funnel is cold audience, bottom of funnel is hot audience, but both are set up with a conversion objective. I prefer to design plans on a marketing objectives basis. At the top of the funnel, your objective is awareness and you're trying to drive reach, clicks, and views—not immediate conversion. What ends up happening to early stage brands that take the conversion-objective-for-all approach is a fast-bloating CAC (marketing objective top of funnel clicks cost much less than conversion objective top of funnel clicks).
An additional thing I do, particularly on the digital advertising side of things, is break out budgets that are intended to drive awareness. A lot of the time, brands are pushing their agencies to optimize ROAS (return on ad spend) and conversion exclusively. But awareness work won’t do that, and it puts their agency partners in a challenging position, since generating awareness campaigns will absolutely drive ROAS down. When you break out the awareness budgets and treat them not as conversion efforts but as overall brand efforts, with KPIs that aren’t conversion driven, you’re able to more accurately consider and build awareness activity without it giving the illusion of pulling down overall digital marketing ROAS performance. I actually have a tool in beta right now that automates this process, because it’s super challenging to do manually with the platforms as they exist, but is so important.
Ben: What's the biggest mistake you see people make with brand awareness?
Jacquelyn: Something that was heavily pushed during the heyday of Facebook ads was the idea that ads could be the primary driver of everything from awareness to conversion. When Facebook was cheap and cheerful, sure, this was a relatively easy thing to hack and tons of brands leveraged this super successfully.
However, this isn't the heyday of Facebook ads anymore. While, yes, advertising has always played a part in most marketing strategies, being over-indexed on paid advertising with a relentless focus on unnatural, immediate conversion, and underdeveloped on all other channels that are designed to create awareness and build a customer journey, has created a P&L mess for a lot of brands as the cost of paid shifted upwards throughout the last year.
Bonus #2: Is NPS a vanity metric?
David VanHeukelom, a serial entrepreneur with 2 exits (both B2B SaaS companies), suggested to me that NPS is a vanity metric. I asked him for more information, because I know a lot of people love NPS.
Ben: Why do you think NPS is a vanity metric?
David: Net Promoter Score (NPS) is a popular customer satisfaction metric that assesses the likelihood of customers recommending a company, product, or service. Although it is commonly used as a performance indicator in business, it has certain drawbacks that make it unsuitable in some scenarios.
Below are a few reasons why NPS might not be the most effective metric to use:
Contextual information is absent: NPS only determines whether customers are likely to recommend a company, without providing any context as to why or why not. As a result, companies may find it difficult to identify areas for improvement in their products or services to enhance customer loyalty.
Limited scope: NPS only gauges one aspect of the customer experience—the likelihood of recommending a company—without accounting for other crucial factors such as customer satisfaction, loyalty, or advocacy.
Relies on a single question: NPS is based on one question, "How likely are you to recommend our product/service to a friend or colleague?" which may not reflect the entire range of customer experiences. For example, a customer may be satisfied with a product but may not feel compelled to recommend it to others.
Inconsistent scoring: The scoring system for NPS can differ from company to company, making it challenging to compare scores between industries or businesses. Furthermore, some customers may find the NPS scale (0 to 10) perplexing, resulting in inconsistent responses.
Potential for bias: NPS results can also be influenced by biases such as a customer's mood or their relationship with the company.
Overall, while NPS can be valuable when used alongside other measures of customer satisfaction and loyalty, it's important for businesses to acknowledge its limitations.
Ben: How can NPS be used successfully by businesses?
David: NPS scores used in conjunction with additional metrics can offer additional data points to evaluate the measurement of customer satisfaction with your product and service. Some of these metrics include:
Net Revenue Retention: Net revenue retention (NRR) measures the proportion of earned revenue from repeat customers and predicts the potential for business expansion from existing customers. A high NRR (over 100%) indicates a propensity of existing customers to renew and expand, with reduced churn.
The Renewal Question: “If your renewal was tomorrow, would you renew?” This question can get right to the heart of your customers current state and willingness to continue to spend money with you.
Customer Retention Rate: This metric measures the percentage of customers who continue to use a product or service over a certain time period.
Customer Effort Score (CES): This metric measures how easy it was for customers to complete a task or resolve an issue with your product or service.
Product Usage: Product usage data can show how users and teams are actively using your application and to what extent, offering detailed insight into the perceived value customers are receiving (or not), from your product.
Ben: What are your final thoughts on NPS?
David: In my experience, I've seen many startups rely on NPS, but without measuring other things, the metric itself isn’t super valuable.
I've seen startups with a high NPS that doesn't translate into growing revenue. But they’re still tracking and celebrating it (which feels like vanity.) And I've seen startups with a low NPS that are still lowering churn and increasing expansion. As a singular number, it's not one I would rely on to really make good decisions on what to do.
Vanity Metrics: The Numbers We Hate to Love
The biggest vanity metric of the last 10 years has been revenue. Unless you can turn that revenue into profit, you have effectively used VC money to buy business.