How We Use Metrics At The Liberators
And how it is helping us get through the Corona crisis
We always tell teams to track metrics that matter to them. For Scrum Teams, this could be cycle time, team morale, and stakeholder happiness. For a marketing team, this could be a conversation rate or NPS. When you have data, you can formulate hypotheses about the experiments you’re doing and when they are successful (or not). When you have more objective data available, you can reduce the impact of social and cognitive biases on what is that you are doing. If all you have to go on our assumptions, opinions, and intuitions, it's hard to be successful.
“If all you have to go on our assumptions, opinions, and intuitions, it’s hard to be successful.”
It would be too easy if we tell teams to do this, and not do so ourselves. So when we started The Liberators in 2018, one of the first things we did was set up an expansive framework to track everything that mattered to us. In this post, we give insight into how we use metrics at The Liberators. And how it prevented knee jerk reactions in responses to the Corona crisis.
When are we successful?
In a previous post, we made the distinction between output-oriented and outcome-oriented metrics. Where the former is mostly about how busy you are (e.g. velocity, hours spent, revenue), the latter is about how effective the outcomes of being busy actually are (e.g. stakeholder value, profit).
It's easy to measure everything and still not know much about what really matters. The challenge of metrics is to pick outcome-oriented metrics that tell you when you’re successful (or not). The first step is to consider what success actually looks like.
“It’s easy to measure everything and still not know much about what really matters.”
So with our mission in mind (“unleashing organizational superpowers“), we envisioned what a successful future would look like. The first thing we agreed on is that we want to support as many practitioners of Scrum and Liberating Structures as we can with our work. This was what got us into the work in the first place. The second thing we agreed on was that we wanted to build a community to provide this support with us. Without a community, our ability to support would never scale. And people might start seeing us as “the experts” instead of realizing that they don’t need experts or gurus.
From a financial perspective, we agreed that success required at least sufficient revenue to cover our costs (including salaries), build buffers to weather future storms, and generate a modest profit. Finally, we agreed that we wanted to create enough space in our agenda to try new things, run experiments, and create valuable free content.
How we decided to measure “success”
With a clear definition of success in mind, we then translated it into indicators that we could measure (our ‘metrics’). For example:
- When we are successfully able to support more people, we should see increases in metrics like the number of followers on various platforms, (re)shares, orders in our webshop, and participants in our various events. In other words; growth.
- When people like what we do and it offers value to them, we should see increases in their engagement with us, our content, and our community over time.
- When we are successfully able to generate enough revenue, we should be able to cover our costs and make enough profit to use that to create space to do work that isn’t paid (like writing, podcasts, research, free apps).
- When we are successfully able to build a community, we should see the number of members of The Liberators Network grow over time. More importantly, if we do a good job, more and more of these people should engage with us and each other.
With a set of indicators in mind, we set up a dashboard in Cyfe and built a service that connects with the platforms we use. At The Liberators, we rely on over 25 platforms to deliver value to our customers and community. For example, we have Buzzsprout for our podcasts, Medium for our blog posts, Shopify for our webshop, and YouTube for videos. These are the obvious ones. Behind the scenes, we use Exact for accounting, Harvest for invoicing, Bit.ly for link-shortening, AppVeyor for deploying our codebase and Fotojet for creating collages. And there’s also use Patreon and Paypal for donations, EventBrite for our events, Trello for coordinating our work and several more.
We collect metrics every hour. They are aggregated once a day. Below is a snapshot of the dashboard at the time of writing:
As you can see, each widget captures one element of how we defined success. We grouped the widgets into activities (how busy we are), output (how much content we generate), outcomes (how valuable that content is), and impact. There are also some critical indicators of class quality and taxes that we need to pay. Let’s explore two widgets in more detail.
Metric: How much space we have
One of our most important widgets is the one that tells us how much space is created by our revenue and profit. The “space” is represented by the yellow bar. The higher it is, the more time we can spend on unpaid work.
During the Corona-crisis, this bar took on another meaning. Because we had to cancel all our paid in-person events, we effectively saw most of our revenue evaporate, save for some modest streams from donations, a few virtual events, and our webshop. At the moment, we’re surviving mostly on our buffers. If we are unable to generate additional revenue, we’ll likely run out by November. Even though that is scary, the good news is that the information from this widget allows us to breathe, consider our options and, and prevent chaotic shifts in order to make some money.
It's also the reason why we’ve more actively started asking for people to support us through a time where our primary source of paid work (in-person workshops and classes) are gone.
Metric: Are people increasing their engagement with us?
When our work and content is helpful and valuable, we expect people to engage more with it over time. So if someone likes our blog posts on Medium, we hope they’ll join one of our meetups, participate in a class and/or support us on Patreon. Although you could simply measure “engagement” by counting the members on our Meetup, or the followers on Medium, this alone is a very indirect measure. So we started looking into ways to capture engagement within and across platforms.
After a lot of experimentation, we developed a service that breaks down how people engage with our work into three groups:
- “Fans” are people that frequently engage with us, our content, or our services over many platforms. For example, they participate in classes, join meetups, order products from our webshop, or even support our work financially.
- “Followers” are people that engage with our work on multiple occasions over time and on a few platforms. They like what we do, but it isn’t super valuable to them yet.
- “Intrigued” are people that have engaged with our work once or twice and on a very limited number of platforms.
Once we identified this break-down and figured out how to count it with some reliability, we calculated the percentages of people that flow from one category to another. The widget shows how many people become “fans” or “followers” from just being “intrigued” (27.3%) and the number of “followers” and “intrigued” that become fans (4.5%). If we do a good job, both should remain the same or increase.
This widget is helpful because it is as close as we can currently get to measuring how successful we are in achieving our mission. The metric is imperfect in how it collects data — there’s much we don’t know — and it’s lagging in that it takes a while to see how experiments affect it.
Trends over time
Metrics on their own have no meaning. You have to interpret them together, and especially the trends over time. Thankfully, we’ve been collecting data since September 2018. Our dashboard allows us to show trends:
How we use the metrics
Barry Overeem and I receive a weekly report with a screenshot of our dashboard. Every so often, we do a What, So What, Now What on the metrics and determine what else is needed from us to adapt to what is happening.
During the Summer- and Christmas-break, we do a more thorough inspection to make bigger shifts in our strategy and start new experiments. For this, we use Strategy Knotworking (as explained in this post). The good thing about having metrics available is that we can create hypotheses about when an experiment is successful. For example, we restructured how we post content to LinkedIn and Twitter in October and expected to see more engagement in terms of clicks and shares. The metrics clearly show that this happened.
If you find yourself reading this post and thinking “Wow, cool, what tools did they use to do this?”, then you’re asking yourself the wrong question. No matter how cool the tools are, they are only as helpful as the data you put in. When we started The Liberators, we asked ourselves “How do we know when we are successful in achieving our mission?”. This helped us to identify metrics that we now inspect frequently.
When we started The Liberators, we asked ourselves “How do we know when we are successful in achieving our mission?”
So if you’re in a team, have a company of your own or work for a company, ask each other how you’ll know you’re successful. It really helps to start measuring that instead of metrics that only tell you how busy you are (e.g. velocity, number of items completed).
We hoped you liked this post, maybe even found some inspiration. Give it a try! And let us know in the comments what happened.