Measuring success by body count is useless

Use the contextual metrics method for smarter wins in event strategy

In 2015, I had the privilege to review over 200 sponsorship requests as the head of Developer Relations Events at Mozilla. Since that wasn’t challenging enough, I also took the role to produce a web dev conference for my company. To stage this first event for success, I partnered with Carly Slater, Head of Growth for Ignite Talks.

Here’s one version of our story:

[sandra] Event organizers around the world submit a form in our system to ask for support. Each sponsorship request is reviewed as an investment opportunity (let’s blame my MBA background for this approach). The input of money plus time plus people had to output something greater than that. In the beginning, the Dev Rel team and I looked at traditional success measures like attendance, engagement level, and getting email addresses. Then we started to feel sophisticated by attaching Google Analytics to measure uptick and create baseline attendance. At the end of year one, we were simply mired in the metrics.

Then Mozilla’s first web dev event, View Source Conference happened, and I was looking at this equation from the other side, as a producer. I had to get back to the basics of asking the fundamental questions of what, who, why, where, and how of the event, rather than simply looking at outcomes. “Why” View Source was answered pretty quickly by our team — to engage web developers and those who work within the community to come together to talk about their pain points and to find solutions they can implement now. The “How” involved having Ignite Talks to literally spark the event — to set the mood, the tone, and incite the conversations we wanted people to have amongst themselves. I found Carly at a local Seattle Ignite Talks event and asked her to join us to orchestrate this movement into the conference opus.

[carly] The first question to ask when planning any event, or even any part of an event is: Why? It’s amazing how many events that are produced don’t have a tangible, measurable purpose. Luckily for me, working with Mozilla, and Sandra in particular, was a textbook example of results-oriented live-experience design.

Of course, that implies that there’s a textbook that teaches modern and effective event measuring practices — which surprisingly, there isn’t.

Numeric vs Contextual metrics: Upgrade your ROI

Let’s start by looking at ROI metrics. Eventbrite recently asked their “valued organizers” to report how they measure events:

  • # of attendees
  • # of attendees had a great time
  • # of the right attendees
  • #revenue $, and that the event ran smoothly

Additional metrics for most events, and at View Source, also include:

  • # of live tweets
  • # of sessions
  • # of celebrity speakers
  • #session reviews
  • #sponsorship $

These are regularly seen in post-mortem reports, and event sponsorship proposals. They seem like solid indicators for how an event is faring, but when we ask the question “what is the purpose of your event?” it’s harder to see any specific, actionable results from these measurements.

Digging a little deeper…

Attendance numbers are great, but if you don’t know what they mean to your business, they’re just a number. Is it better to have 100 unique tweets or 2000 retweets? The answer is contextual to your event and your event’s deliverable purpose.

The Mozilla team knows that a sold out event of disengaged attendees is not nearly as successful as a half-full event with attendees who feel connected and well networked. Having seen Ignite events in action locally, they saw an opportunity to kick off their conference by directly connecting their audience, allowing their attendees to get exponentially more out of the days that followed.

[sandra] The team also worked on content quality for all aspects of the event. Working with Carly, we honed the overflowing list of speaker submissions for the Ignite Talk that fit the purpose of the event. Having this as our main parameter, we were able to then select the talks that fit the program cadence perfectly.

[carly] Mozilla’s “why” is based on contextually measured results. That means they’re tracking the difference between “showed up” and “engaged”. During the planning of Ignite View Source, Sandra and I started by first identifying Mozilla’s desired end results from the conference, and then determining how to measure and report on how Ignite View Source was delivering on each metric.

We settled on some key pre-event results that were important to the projected success of the event, using a variety of metrics that would allow us to report on very specific success markers. The secret to this sauce is that the success markers were chosen by the Mozilla team themselves. When it comes to online traffic, for instance, Sandra knows that the difference between “lots of tweets” and “a small but strong showing of authentic tweets with keywords and phrases that indicate trust, excitement and anticipation” is vast and game-changing.

[sandra] And finally, we looked at the revenue targets in terms of sponsorship and ticket sales to serve as the proof of concept that our community truly found our efforts worthwhile. Because this was our first conference, we tempered expectations to fit reality: most events on year 1 operate at a net loss. However, by thinking of the event as a product, the loss can be classified as part of sunk costs.

Change your perspective on success

So, to create an intelligent design to your event planning, start with the basics, then set your measurements. Look at the event as a product, not an activity. If you can’t answer “why” and “how”, then all the “organization” in the world cannot deliver results.

Do you have a marketing strategy? Do you have content? Or are you part of the “me too” syndrome, following your competitors, instead of truly being competitive in the market place?

For the View Source Conference, starting with the Ignite Talks, we achieved our stakeholders’ KPI to put a stake in the ground as the first neutral platform for a web developer summit. Second, we created the right platform to encourage the conversations about the progress of the web. And perhaps most importantly, our team created a program that continuously added value to the overall event: Ignite Talks to spark the conversation; keynotes that addressed topics raised during those conversations; discussion corners where deeper details within those conversations were explored; socials to relieve tensions from the momentum of conversations; and closing talks to seed future opportunities for conversation.

Think about these new set of measurements

To define your event’s engagement plan beyond tshirts, pizza and beer consumption, create a value proposition matrix — to attendees, to company, to community — and evaluate who benefits most from the event. Determine if that helps support and grow your top line goals. Also, fill in the knowledge gap of how to properly grow your event by looking at collaboration opportunities with strong partners.

In the past, attendance has been used as a primary metric, but we now know that measuring these unqualified results is just a sandtrap. We now evaluate events as a product, rather than a one-time “activity”. All of the “organization” in the world cannot deliver results if you’re not using contextual metrics to measure Why and How. When you start asking more of your measurements, you’ll start getting more out of your event.