Guest Speakers: Colin Holtz, author of “Beyond Vanity Metrics” & Bhavik Lathia, Color of Change
MobLab Live: May 14, 2015
In trying to create social change, we often ask ourselves: how do we know what works? This simple question can be alarmingly tough to answer. Typically in online engagement strategies, we measure our member’s engagement with petition sign-ups, list sizes and followers on social media. These “vanity metrics” often sound big and impressive but can lead to flawed decisions that doom membership-driven organizations. So to find alternatives, the Mobilisation Lab at Greenpeace and Citizen Engagement Laboratory released a report called Beyond Vanity Metrics. Joining us on MobLab Live as the report author and experts explore a summary of their findings. Watch the conversation and find additional resources below.
Key Learning: 5 Pitfalls of Vanity Metrics
- Flawed understanding, leading to poor decisions
- Short-sighted decision making
- Bad staff incentives
- Failure to engage members, leading to unsubscribes and people tuning out
- Organizations that don’t live up to potential
Key Learning: 3 Lessons
- There is no “perfect metric.” Every single metric has flaws or biases your thinking in some way.
- Your mission should define your metrics. The best metric comes from a deep, shared understanding of the organization’s theory of change.
- Get the sequencing right. For example:
- Define your organization’s mission.
- Understand what it will take to achieve that mission.
- Identify key metrics for measuring progress towards mission.
- If necessary, identify and track “leading indicators.”
- Regularly assess your metrics, and don’t be afraid to change them.
Key Learning: Alternatives
There are no “perfect metrics” but some starting points include:
- Cohort Analysis: Rather than looking at all members as one unit, breaking into cohorts of people who share a common characteristic over a certain period of time (i.e. # of actions members have taken).
- Mission-Centered Metrics: In certain situations a raw, aggregate number can still be useful if there is a clear connection between volume and a desired outcome calling them “mission-centered metrics” (i.e. # of people who attended a rally or “total victory experiences,” the # of members who have been a part of a winning campaign).
* This not a comprehensive list nor are external resources endorsed by Greenpeace.
Live Q&A with #BeyondVanityMetrics
A snapshot from Twitter of the conversation and questions on this MobLab Live conversation:
All metrics have flaws. Pick ones that incentivize the right behavior. #BeyondVanityMetrics
— Jo Miles (@josmiles) May 14, 2015