016 – The Vanity of Nonprofit Metrics

Sometimes, people latch on to poor metrics when they can’t find a good proxy for what they want to measure. And those poor metrics can lead them away from what they want to accomplish.

In this solo episode, Spencer challenges the popular digital metrics of traffic and stickiness and explains how to find your “smoking gun” metrics instead.

• Article: The Vanity of Nonprofit Metrics
• Article: The Surprising Truth About Digital Metrics for Nonprofits
• Service: Health Nonprofit Digital Diagnostic
• Thoughts or questions? Email Spencer

Full Transcript

Spencer Brooks 00:03

Welcome to Health Nonprofit Digital Marketing, we’re a podcast for nonprofit marketing and communications leaders using the internet to reach and engage people with health issues. I’m your host, Spencer Brooks of Brooks Digital, a digital agency for health nonprofits.

Spencer Brooks 00:20

Today, we have a special solo episode titled the Vanity of Nonprofit Metrics. So, there’s also a companion article on the Brooks Digital website for this podcast. It’s titled exactly the same as the episode: the Vanity of Nonprofit Metrics. And so, this is just a little bit of a deeper dive into that subject. If you are more of an auditory learner, then this might be helpful. This also gives me a chance to express some of my thoughts in a longer form, as opposed to the article. But feel free to check it out on the website if you want the written version, or if you just want to have something to refer to along the way. So the central idea behind this topic is that sometimes people will latch on to poor metrics when they can’t find a good proxy for what it is that they’re trying to measure. And then those poor metrics can lead them away from what they’re actually trying to accomplish in the first place. So I think a good illustration of this is in the software world. And of course, my background is, once upon a time, I was a software engineer, so this hits kind of close to home for me. But in the software world, there’s this controversial management technique, where a manager will measure the productivity of a software developer by the lines of code that they might write, in a day, or in a week, for example. And that might seem like a good idea initially, right? You’ve heard the maxim, what gets measured gets managed, and so a good manager is going to try and find something to measure. There’s nothing wrong with that. But where that gets into trouble, sometimes in the software world, is that measuring a developer’s productivity, in terms of the lines of code that they write is very similar to measuring a writer’s productivity in terms of the number of words that they write, or the length. And as I’m sure you know, the quantity of writing doesn’t necessarily have anything to do with the quality. And so, what ends up happening is that managers who tend to measure lines of code, get a lot of code, which tends to lead to bloated or overly complex or slow, inefficient software. And that is a, it’s an unexpected outcome, because they’re not shooting for that right. No one’s shooting to write bad software. But the difficulty comes because defining what good software is, is extremely difficult. It can change depending on the organization. Sometimes good software, for example, is something that is released on time to meet a certain market demand, even if it’s not quite perfect. And other times that can mean actually taking a lot of time to develop a piece of software, so that it is going to compete with some behemoth that’s in the space. And so, because that’s hard to measure, a manager will often take a proxy for that, that is easier to measure and then end up with a not so great result, actually, sometimes even the opposite of what they were trying to achieve.

Spencer Brooks 04:03

Another example of this, that I saw recently is there’s this leader in India, and they said, too many people were dying from venomous snakes. So that leader offered money to anyone who brought them a dead snake. And what they didn’t anticipate to happen was that people actually started to breed venomous snakes in private so that they could kill them and then bring them to the government. Again, this is another unintended consequence of trying to put together a measurement that is not a close proxy to what it is that, in this case, this leader was trying to accomplish, which was to reduce deaths from venomous snakes. They ended up actually getting more venomous snakes because people were breeding them. And so the that’s the idea behind this is that oftentimes when you’re measurement is a poor proxy of your goal, it can end up leading you in the wrong direction. And nonprofit website metrics, they’re no different. I think many organizations, they tend to be obsessed with metrics such as traffic or stickiness. Sometimes they’re called vanity metrics, right? They’re these big numbers that are impressive. They’re pretty easy to measure. And they’re easy to obtain. Right, you can just log into a dashboard somewhere, and you can go grab that information. It’s also impressive, it makes you feel good, right? But presumably, many people think, okay, these are kind of a close proxy to our mission, if more people are visiting our site and staying there longer than we’re helping more people. But BuzzFeed, for example, they get a lot of traffic and a lot of stickiness, but they’re really doing nothing to improve people’s lives. And so, now, I’m not saying that traffic and stickiness aren’t necessarily bad metrics. But what I am saying is that often they are poor proxies of your mission. So, I guess the thing about this is that the metrics have become widespread, this idea of traffic and stickiness, I think, for a couple reasons. Because you might be wondering, okay, if these are poor proxies of my mission, then why do so many people talk about these metrics? Why are they so popular? And I think that they’re popular for a couple reasons. These are some of the ones that I can think of, I’m sure there’s more. But the first, this is kind of an interesting one. So, if you start to read around on the internet, about what metrics you should be tracking these numbers come up fairly often. I think this is in part because a lot of the marketing numbers or even the marketing tactics that are used are, they’re put into practice by these digital behemoths like I’m thinking about social media companies, or the media and publishing sphere, that tend to focus a lot on traffic and stickiness, and you’ll see them shaping their products around that. And these are things that we use on a regular basis. And they shape what we expect from a website and from the internet. And we tend to copy as people the things that we like, and the things that we’re familiar with. And so, what these big behemoths are doing, we naturally will follow them. And other people will write about that. And those things, they tend to show up in a lot of places. But the deal is that a lot of these digital behemoths are optimizing around traffic and stickiness. Because those things drive ad revenue, that’s how they make their money. And I’m venturing to guess that that is probably not your primary mission is to drive ad revenue. And so it helps to understand, certainly, that a lot of the experiences that we see on the internet, and even in many cases, a lot of the metrics that are paraded around the marketing world, tend to, in some cases, be focused on goals, such as ad revenue, that may not actually be anywhere near what your goals are, as a mission driven organization. So that’s something interesting to think about. I also think that, honestly, funders are asking about these metrics, too, even though maybe they may not be great proxies of your mission. They are still appearing like, you know, if you’re doing grant writing, or if you need to prove to a funder that you’re showing some kind of impact, it’s better to have something than nothing. And so I think while funders, they’re getting smarter about the data that they’re asking for, they’re also asking for metrics that they’re maybe easier to easy to manipulate, or they may not even have necessarily a direct proxy to your mission. But I think nonprofits are incentivized to boost those metrics and pump them up because they look good for fundraising purposes.

Spencer Brooks 09:31

And I think lastly, that nonprofit technology, it also tends to lag, say three to five years behind the for profit sector. And in many ways it tends to take whatever’s working as kind of a safe bet, like I don’t necessarily see a lot of innovation in technology coming out of the nonprofit sector. I see more of some ,derivative is not the exact word that I want to use here, but it tends to be copying the safe bets that have been tried and tested in the for profit technology world over the last couple of years, and then pulled into the nonprofit sector. And I do think that the for-profit sector has become a little bit better and sophisticated about measuring different types of data or things that are just beyond traffic and stickiness and some of these big other vanity metrics. But maybe the nonprofit world is not not quite caught up to that, as well. So I think what’s important, again, to mention is that, I said it before traffic and stickiness are not bad metrics, if your goal is to directly measure and to boost your online awareness, or your website engagement, right, if you’ve set out to say, Okay, I’m intentionally trying to boost online awareness, or I’m intentionally trying to increase the engagement of visitors on our website, say for its own sake, or if that’s just part of your strategic plan, then I do think traffic and stickiness are these different metrics are actually very, very helpful. Because they’re measuring exactly that. But I think what gets people into trouble is when they try to extrapolate that out all the way to their mission and say, Okay, well, more eyeballs on our site means that we’re helping more people, well, it’s like, more eyeballs on your site, kind of, you know, that might mean a lot of things. They could be getting help, you could be improving their lives, or you could be doing nothing at all. And so I think if you’re convinced, for example, that traffic is the best way to measure your impact. It can be easy to do things like artificially inflate your numbers with digital ads, or pour a bunch of money into SEO. But if those people stay on your site for 10 seconds, and then they leave forever, have you really helped them? Are you accomplishing your mission? And so, I think that that’s, that’s a lesson that we can take from this broader idea of people latching on to metrics that are not necessarily a good proxy for what they’re really trying to accomplish. But when there’s nothing else, what do you do? So, I think the best digital measurements, at least in my opinion, they’re more closely linked with the outcomes of your mission, and you can’t necessarily copy them directly from or other organizations, or you can’t necessarily Google, what are the best measurements for a nonprofit website, and get a list of metrics that are going to be extremely closely tied to your mission. I think that the measurements that are going to be really good proxies of your mission are going to actually have to be unique to your organization. And it’s why I think that a lot of organizations either just, they don’t maybe think about it that way, or they don’t put in the effort to really think about, okay, how would we go about measuring from a digital perspective, the impact of our work, but these digital measurements that are super close to your outcomes, I like to call these your smoking gun metrics. And I think they’re really worth focusing on.

Spencer Brooks 13:40

So, here’s a scenario for you, a thought exercise right to get a more practical idea of what this might look like for your organization. So I want you to take a second and imagine right now that a funder approached you today, and they offered to write you a blank check, if you could prove beyond a shadow of a doubt that your digital presence was contributing to say curing Alzheimer’s or helping people with heart disease, increase their lifespan, whatever your mission is, if you could wave a magic wand and have any data, any numbers that you wanted, what metrics would you need to convince them?

Spencer Brooks 14:27

That’s a very interesting question. And because I think that oftentimes the answer to that question is not a number that you can go cherry pick from a dashboard somewhere. Many times the answer to that question is not logging into Google Analytics and looking at your report somewhere, it’s actually data that you don’t have it’s maybe data that you don’t even know how to get. But it’s really important to ask yourself that question, because instead of hunting and pecking through a Google Analytics stuff dashboard are trying to install 1000 different analytics tools to just see if it produces something interesting for you. It actually gets you thinking about, okay, what is that smoking gun metric? A lot of times, that data ends up being more qualitative instead of quantitative. So, you end up needing to collect it in a different way than just installing, you know, some analytics software on your site or something like that. So to give you some examples of what that might look like, it could be that that smoking gun metric might be 84% of our website, visitors leave feeling more hopeful about their ability to care for their child with autism. Like, that’s great. That’s if you’ve had that data. And your mission is to, you know, to provide hope to people or, you know, parents with children that have autism, then if you have the data to say, Yeah, this percentage of our website, visitors leave the site feeling more hopeful. Like that’s smoking gun metrics right there. Like, there’s proof.

Spencer Brooks 16:10

Another one could be, the average person has lost 21.2 pounds since they began reading our online educational materials. Like if you’re, for example, focusing on obesity. And again, that’s like that’s very, very close to your mission. And what would that do for your organization? If you were able to answer that question? How would that change your fundraising, you have the ability to go out to a funder and say, hey, look, these are our numbers. This is what we’re trying to do. Do we have your support? How would that change, writing your annual report for example. How would that change your ability to find and recruit good staff members, if you can pull out that kind of information? And I think certainly in my experience, those are the kind of nonprofits that are able to tell really good stories, to have compelling messaging is when they can actually produce these numbers that are very closely tied to their mission. But my suggestion is that oftentimes those numbers are not arrived at by going and trying to find them somewhere, that they already exist, oftentimes, they have to be invented before they can be collected.

Spencer Brooks 17:41

And so, I’d encourage everyone listening to think about what is your, your smoking gun metric, or metrics and, and write those down and think about how you might go about collecting them. So, and actually, on the topic of collecting those metrics, there’s a, there’s a couple practical ways that I think you can do that. These are specifically related to your site, right? That’s certainly my specialty. That’s where we live at Brooks digital. But some ways that we can go about that, a as nonprofits with a digital presence is things like on site surveys, for example, there’s a tool called Hot jar that will allow you to, for example, ask a visitor a question. Like, you know, it could be after reading this article, did you leave feeling more hopeful, or something like that. And whenever someone visits the page, like maybe halfway down the page, if they’re reading an article, you can pop that up and get feedback. Another interesting way to measure this is NPS. There’s some different perspectives on whether NPS is a great, it’s Net Promoter Score, by the way, for those that aren’t familiar with that acronym. And how that works is that, it’s the question that I’m sure you’ve seen everywhere, like how likely on a scale of zero to 10, would you recommend this service or this product to a friend, right? And so it does some complicated math and groups people into promoters or detractors and kind of it gives you a sense of how well you’re doing based on essentially that single question of how likely is someone to recommend this? And so, you know, I wouldn’t, I don’t know if I could go so far as to say this is the single metric that you need to optimize your entire site around. But what I do find it to be helpful in doing is that number one, if your mission is say, related to making someone’s life with a chronic illness, better, and part of that is providing, say online educational materials, then you could ask a question such as how likely are you to recommend other our resources to someone else with your condition? And then they’ll give you an answer from zero to 10. And then you can follow up with a question, what, what, why did you give that answer? And so if they scored you really high, then you’ll get a really good sense of the things that you’re already doing that make this resource more recommendable, and if they score you low, then you’re gonna get a good sense of, here’s what we need to do to get better and to improve. So that’s another it’s another interesting metric that you can track. Sometimes even feedback buttons on the site can be helpful. Like for a client, we recently did a, a simple button that says would you like? Do you recommend this article that kind of similar the NPS thing to be honest, but it’s more of a kind of a social proof idea where someone reads it, and they just can click a button to thumbs up it. And that’s pretty powerful, because then you can get a sense of, Okay, how many people recommended this article gave it a thumbs up. And that that’s, that’s useful data. So I think there’s different things that you can do. But it starts with this idea of thinking about those smoking gun metrics, and realizing that you can’t start and you can’t use a poor proxy, you can’t use metrics, such as traffic or stickiness necessarily, to measure your impact, but you need to do the work to get a little bit closer to whatever your mission is.

Spencer Brooks 21:50

Ironically, though, traffic and stickiness can actually end up contributing or being sort of these lead measures of your bigger metric, your smoking gun metric, but they can’t take the place of it. So for example, if your metric is okay, we’re measuring the percentage of our website visitors that leave feeling more hopeful, then you might start to look back through your data and say, Okay, is there a particular source of traffic that is finding our content more helpful or more hopeful, right? Then you might be able to go and prioritize that. Or you might be able to learn, okay, well, you know, we’re not really getting, or maybe people that just aren’t a good fit, we’re targeting the wrong people on this social media platform or something, it’s actually helpful to be able to dig in and understand, where are you reaching your people or where are you not, or literally just to say, Okay, we’re doing a great job, a high percentage of people are finding our content helpful, for example., and so we’re just going to literally scale this by adding more traffic, that’s great. But, again, it all starts with having that smoking gun metric, and then these other lead measures can factor into that.

Spencer Brooks 23:22

So that’s some food for thought for you. Again, if you want to read this, the companion article, this is up on the Brooks digital website. It’s titled the vanity of nonprofit metrics. That’s a written version, probably, uh, the CliffsNotes to this, although there’s some things in there that I may not have covered here. But that does wrap up our show today. I hope you enjoyed the solo episode. We are a new podcast, I say this every time but it really does help when, whenever you’re able to rate or review us on Apple podcasts wherever you end up listening to this. It makes sure that other people can find this content if it is helpful for you. And this show is also a part of the thought leadership of Brooks digital. We’re a digital agency for health nonprofits, and we specialize in web strategy, design and development. So if you liked this podcast again, you can visit our website, Brookstone digital, you can find more of our insights there and learn about our work.