Written by Rick Mans

About

Vanity Metrics – Measuring the wrong thing

About the time this blog is published I have just finished my presentation at B2B Goes Social. At this event I presented about Getting Real with B2B Communities and to be more exact about the pitfall of using vanity metrics instead of real actionable metrics. The main reason to do a talk about B2B communities and vanity metrics is because this was also a topic I did 2-3 years ago for B2C communities. Since B2B lacks 2-3 years behind B2C I found this topic rather applicable for reuse in a B2B context.

These are the slides I have used and below the slides there is a summary of what I have talked about.

With communities there are two general mistakes made in approaches:

  1. The 90-9-1 rule defined by Jakob Nielsen. Read why I think this limits the efforts of community builders here.
  2. Vanity Metrics.

Vanity Metrics

Vanity Metrics for me have two main appearances: aggregated numbers and false, non actionable metrics. For the latter I choose to dive into the aspect of signing up users into your B2B community, since one of the key KPIs I see is the number of users and since getting users on your community is important though the number of users is irrelevant, it is about what these users are doing.

Aggregated numbers

A typical view on success of communities is the total number of views or the total numbers of posts or to abstract it a bit: the total number of actions in a certain time frame. These give false information. Since even if your community has an increase of 200% in the number of views, is it really great? First of all it is important to know what has caused the change and second it is important to zoom on a more detailed level to see what the change actually is.

For example if your community was linked on Reddit, Slashdot, Digg or Twitter and these channels drove an enormous traffic spike then it is nice to have this spike, to there is no sustainable change. However if you have made a change in the user interface and all of a sudden your users don’t visit two pages per session but 6 pages per session you have a more meaningful insight.

Detailed Analysis

Even more meaningful would be to divide your users in different groups, such as new versus returning users, but also based on the life cycle of the user and perhaps even on demographic data or based on typical actions (users that buy your goods, users that review them, users who just read information).  In this way you can see what happen to certain changes. Also if you average user has a life cycle of three months in your community and you have an enormous peak of sign ups, you will know there will be a huge drop-off of users in three months after that and you most likely want to be ready for that.

 Attracting users

Even though the sheer number of users would be a vanity metric, it is still important to attract users. Since without users that do something communities are  boring. Research of Janrain has shown that users on different media have different preferences on (social) sign up, however that doesn’t mean there is just one solution that fits all. Important for social sign in and sign in-flows in general is that these are A-B-tested continuously. Since your first sign-in/up-flow is based on the assumptions you have about how it would work best. This is not about what the user thinks that work best. So involve the user early on, either by doing A-B testing, user interviews or any other way, though it is about validating the assumptions you have early on. There is no such thing as ‘the user’, users are people too.

Going viral

‘How do I get to the tipping point?’, a question often asked, since if you manage to get at a tipping point growth is happening at an immense rate. The basic premise to achieve this with the sign up of users is to make that every new user will bring more than 1 user with them. If every new user will bring a least two users you will get to a tipping point. To get there are several techniques you can use. You can choose the more black hat approach such as automatically spamming customer network with invites for your community, however this might drive traffic on short-term, it most likely will drive your reputation into the ground.

Therefore providing relevance is more important. Research from Cornell University about Facebook has shown that relevance is also something worth to A-B test:

“What jumped out at us was that someone’s likelihood of joining really corresponded not to the number of friends represented, but to how many disconnected groups the friends listed on the e-mail fell into,” Kleinberg says. “We were surprised by how clean the effect was.”

Getting in touch

Standard approach of communities is to get in touch twice with new members: one time when they join and the other when somebody leaves. However it might be more valuable to get in touch more often and first all not at the first moment they just joined. At that moment in time you have their attention, it might be more valuable to get in touch after a few days when they forget about you.

Flickr, or to be keep it more human: Caterina Fake and Stewart Butterfield, founders of Flickr, used to welcome every new member on Flickr personally. This is a tedious job and consumes a lot of time. However each interaction with your first users is well spend. It is an investment you’ll have to make to make sure that your community will be viable. It is an investment in creating your first few community evangelists who will promote your community for your. And it is an opportunity to get instant feedback from those who are willing to join your community in this early phase.

The best solution for you

The best solution for you, is the solution that works best for your customers / users. Therefore don’t assume too much upfront. Start early, (AB)test early and adjust often. Don’t use vanity metrics, use real actionable metrics that can be related to your business goals and your assumptions. Validating your assumptions early is important for your community, since if you hold on to incorrect assumptions for too long your community will probably not bring what you had expected.