MobLab Live: Beyond vanity metrics

In trying to create social change, we often ask ourselves: how do we know what works? This simple question can be alarmingly tough to answer. Typically in online engagement strategies, we measure our member’s engagement with petition sign-ups, list sizes and followers on social media. These “vanity metrics” often sound big and impressive but can lead to flawed decisions that doom membership-driven organizations. So to find alternatives, the Mobilisation Lab at Greenpeace and Citizen Engagement Laboratory released a report called Beyond Vanity Metrics.

Joining us on MobLab Live as the report author and experts explore a summary of their findings. Watch the conversation and find additional resources below.

Key Learning: 5 Pitfalls of Vanity Metrics

1. Flawed understanding, leading to poor decisions
2. Short-sighted decision making
3. Bad staff incentives
4. Failure to engage members, leading to unsubscribes and people tuning out
5. Organizations that don’t live up to potential

Key Learning: 3 Lessons

1. There is no “perfect metric.” Every single metric has flaws or biases your thinking in some way.
2. Your mission should define your metrics. The best metric comes from a deep, shared understanding of the organization’s theory of change.
3. Get the sequencing right. For example:
   1. Define your organization’s mission.
   2. Understand what it will take to achieve that mission.
   3. Identify key metrics for measuring progress towards mission.
   4. If necessary, identify and track “leading indicators.”
   5. Regularly assess your metrics, and don’t be afraid to change them.

Key Learning: Alternatives

There are no “perfect metrics” but some starting points include:

1. Cohort Analysis: Rather than looking at all members as one unit, breaking into cohorts of people who share a common characteristic over a certain period of time (i.e. # of actions members have taken).
2. Mission-Centered Metrics: In certain situations a raw, aggregate number can still be useful if there is a clear connection between volume and a desired outcome calling them “mission-centered metrics” (i.e. # of people who attended a rally or “total victory experiences,” the # of members who have been a part of a winning campaign).

Additional Resources:*

1. Slides by Bhavik Lathia (Color of Change)
2. Beyond Vanity Metrics report

* This not a comprehensive list nor are external resources endorsed by Greenpeace.

Live Q&A with #BeyondVanityMetrics

A snapshot from Twitter of the conversation and questions on this MobLab Live conversation: