Designing Analytics for Growth Measurement
Lessons learned from years of analytics driven thinking and growth-centered product management.
At Marco Polo we use Amplitude (love it!), and some of the concepts use their terminology but can be applied to any analytics tool that deals within events and properties. My experience is also social-network-centric, but my hope is that most of it is applicable to any problem domain with some fitting.
Events
There are two tiers of events we record:
- KPIs — these are your critical events, there’s no question about recording all of them. Records, signups, messages, etc.
- Actions — what’s a user doing, which very specific buttons and other UX elements pressed on the way to a KPI. We’ve found these very helpful in analyzing our signup flow (.e.g name entered, email entered, next clicked). When something goes wrong and users aren’t converting, you can create a funnel between any two events. These events can be sampled as as necessary.
Event Properties
Amplitude allows you to further associate key/value pairs called properties with each event . Like with events, we divide these into different classes:
- Contextual — the relevant state of the user/client/appilcation at the time of an event. Look around in the code at what influenced the logic tree that brought the user to this point. Also, what version of the client/server is processing the action? What’s their OS version etc? That can be very useful in watching roll-outs of major new features. For mobile apps it can be useful to add a property for carrier/network to tease out issues that might be.
- Environmental — if the user is performingan action, how many times have they done this action in total, this week, etc. Are they doing this action as part of a distinct UX flow? Who’s the recipient of this action? What are their usage patterns?
- Experimental — make sure that every event contains the full context of special-cases that might affect their particular UX, experiments being a huge contributor to that.
- Global — things at a user or system level unrelated to the current action. Is this user a “great user” based on whatever that means in your system? Property it. You’ll then be able to create reports about usage by your best and worst users and segment other behaviors. Counts can also be super useful to discover how many of each of the important objects in your app users have created (groups, conversations, etc).
Some very specific example properties that have served us well at Marco Polo:
- DaysSinceLastACTION — this is the number of days since the user last performed a particular action. From this we’ve done entire studies on resurrection and what brings users back as it gives you a perfect start event to analyze funnels or retention.
- DaysSinceSignup — we like to segment reports by new/old/tenured/tenured+something-else. Recording the raw days since signup allows us to define those segments dynamically and look back at long-term trends.
- L7RecordDays — how many out of the last seven days including today that the user recorded a message. We segment this out to define “daily users”.
- Source — what caused the user’s original signup.
“Next Generation” Properties
Not Star Trek here, just propagating properties along to invited users. What we do is take all of the user properties we set for everyone and add them to users who sign up from an invite generated by a user with a prefix. So ‘CalendarDaysSinceSignup’ of the user who invited a user becomes ‘SenderCalendarDaysSinceSignup’ on sign up.
Re-basing Your Analytics
Classically you’d have a single “project” or table tracking each user’s events in a stream keyed by their UUID. Consider taking that a step further and have additional Amplitude projects using non-user keys. Our most useful secondary project is keyed by conversation-id and allows us to track conversations as an entity to learn about their health.
Sampling Events
Some events or even classes of measurement will overwhelm your database or budget and can be sampled. Some tips:
- Sample on your primary key. If all your analytics are generated from a UUID perspective, hash that mod 100, and sample a subset of that.
- Create an un-sampled cohort where you record 100% of events and flag it with a property (we use AllEventsRecorded, boolean) even on un-sampled events. This will allow you to still create funnels that start with sampled users where you’ll know all follow-on events are recorded.
- Include a property that indicates the sampling percentage on only sampled events. This will let you back it out for top-line reports programmatically while adjusting to changes in sampling — and save your sanity when you can’t figure out why volume on an event is so low.
- Un-sample events for users included in experiments. Warning though, this will change totals all the time for events hit by users in the experiments unless you use the AllEventsRecorded cohort.
Conclusion
There’s a ton here and it would take a lot of discipline to develop and integrate into your analytics flow. But I’d argue that every day that goes by not gathering historical data with these events and properties is a loss. Weight the cost/benefit/loss for yourself and integrate when it’s appropriate for your business!
Author
Casey Haakenson lives in NYC and is the Head of Growth Product Management at Marco Polo, loves to talk all things startups/business/tech/growth, and can be reached on gmail.com as millisecond.
Reviewers and Contributors
Many thanks to Michal Bortnik and Nick Nugent for your ideas and reviewing early drafts!
