Stop Listening To Your Superfans (If You Want Real Insights)

Listen to this story and tell me when you spot the problem:

Let’s say I’m the type of guy who’s invested in personal development (I am). I want to get feedback on my strengths and weaknesses so I decide to ask a few people I know for input.

I flip through my phone and find the names of some people who know me well—my mom, best friend from childhood, that guy from work who cares way too much about Game of Thrones and I really regret ever bringing it up in the first place—and shoot them an email with a few questions about my “superpower” and “areas of growth” or whatever.

I get their answers back the next day and guess what? I am generally a pretty swell guy that isn’t perfect, but I’m not doing too bad. I conclude that I basically need to do more of the same stuff and maybe brush my teeth more.

Okay, so I lied. There are actually two problems here. Did you catch them?

1. I gravitated towards gathering feedback from the people I had the most access to.

2. Those people already liked me, had a personal relationship with me, and definitely were not 100% honest because they didn’t want to ruin our relationship.

Like me, your organization has its superfans too. And holy sample bias, Batman, you need to stop listening to them.

I understand your motives, but here’s the thing

If you feel like I just reached through your screen and pointed my finger at you, you’re probably aware that your feedback is biased. But what else are you going to do? Some feedback is better than none, right?

Well, kind of. It depends entirely on how you collect and use it.

Many of you are you collecting feedback via some method of convenience sampling:

• Distributing surveys individually to your professional members or active program participants;

• Soliciting your email list;

• …or collecting what I will gently refer to as “organic” feedback from people who call, email, or post emoji-laden praise on your Facebook page.

The people most likely to give you feedback this way are highly engaged with your organization and do not represent a balanced sample of your entire audience. When the feedback you get is centered around the desires and needs of the highly engaged users of your site, it’s missing a lot of hidden negatives that would be expressed by less-engaged users.

This feedback is useful, and there is something to say for how quick and easy it is to collect. But you need to be really, really careful when using it as the basis to improve your digital presence.

Getting at better feedback data

Okay, so you get the point. Let’s talk about how to approach this in a different way.

I want you to think about your online audience in two buckets:

1. Your existing website users;

2. The total potential audience you could conceptually reach (i.e. your Total Addressable Market / TAM).

The reality is that some of your website visitors will not be a part of your TAM, but there should be a significant overlap:

At a high level, you’re shooting to gather feedback from:

1. The segment of your online audience that’s part of your TAM (the striped section inside the “Online audience” circle above); and

2. Members of your TAM that have never visited your website before.

Let’s go over each group in more detail. I’m going to offer some suggestions on how to reach them, but there is an entire UX discipline dedicated to this stuff. However, if I’ve helped you begin thinking about your online audience in a different way and recognize where you need to change how you collect and use feedback, that’s a win.

#1: Members of your online audience

I want to start here because these are the people you have been convenience sampling—you just need a better way to reach them.

One way to reduce that sample bias is to use a popup survey tool on your website such as HotJar (which is free for nonprofits, by the way):

Source: hotjar.com

While this won’t totally eliminate sample bias, collecting participants directly on your website will give you a wider, more representative sample of your audience, which gets you more accurate feedback than surveying your superfans.

To make this even more effective you can offer a $5-10 Amazon gift card to those who participate. I would suggest you add qualifying questions to screen participants, especially if offering compensation, to ensure you are getting members of your TAM and not a random person who wants the free gift card to buy a toaster on Amazon.

#2: Potential audience members who have never heard of you

The other half of the equation that’s often neglected are the “unreachables”—those folks you can help who have never heard of your organization.

I’m sure many of you would love to hear an ideal constituent narrate their opinion of your website the first time they see it. How much valuable insight would that produce for your organization?

In many cases, this is possible with tools like UserTesting. You can target a specific audience segment (for example, we have helped clients survey people with a particular medical condition) and do a live interview with them or have them record themselves performing a set of tasks on your site.

If you are targeting a more niche audience, you can also hire someone to recruit participants on your behalf. Depending on the type of research you’re doing (e.g. usability testing vs. gathering statistics), you may only need a minimum number of participants ranging from 5-20, which is very manageable.

These “unreachables” will also offer you valuable insight on how to, well, reach them—e.g. how they research symptoms or the type of information they’re searching for. This can help you craft a better SEO, social media, or digital advertising strategy to get your organization in front of more eyeballs.

In short, doing the proper research on your TAM allows you to build smarter strategies that grow your “online audience” circle:

But all of this starts by collecting feedback from the right people, in the right way. Stop just listening to your superfans, and make sure you’re giving everyone else a seat at the table.