Apr 18, 2016
How One Jira Ticket Made My Employer $1MM/Month: 7 Metrics that Actually Matter
“You only get what you measure, so be careful what you measure.”
I spent the first half of my career consulting with all kinds of businesses on how to leverage digital presence (meaning apps and websites) to move core business KPIs. My sales pitch was simple: Put me 100% in charge of the engineering org, and I’ll measurably improve the bottom line, or you don’t pay.
I didn’t do it by adding features. I did it by focusing engineering resources on the metrics that matter. That laser focus over the span of a decade or so, with all shapes and sizes of business taught me a very important lesson: Pay attention to the right numbers, and forget that other numbers exist at all. Don’t get distracted by noise.
Maybe you’ve heard “what gets measured gets managed”, but that’s only half the quote:
“What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.” ~ Peter Drucker
In other words, it’s very important to measure the right things — and even more important not to measure the wrong things.
I have some bad news for you. If you’re like most software development teams today, you’re measuring the wrong things.
Why is Agile a Dirty Word?
I recently saw yet another popular blog post condemning “agile”. But the author wasn’t talking about agile. The author was talking about scrum meetings, ticket counting, and velocity-measuring-micro-management.
Viewed in the aggregate, a steady stream of closed tickets looks like work getting done. Granted. To derive any business value, you must complete tasks, so, number of tasks = amount of value delivered. Right?
Oh, no, no, my friend. You could not be more wrong.
Several years ago I was working for a world-leading retailer on a shopping cart. One day I stopped writing code and closing tickets in Jira (the ticket tracker of choice in its day). I added one ticket: “Usability study.”
We’d been working on a redesign of our shopping cart for more than a year, and the time for the new release was quickly approaching. Up until this point, we had not conducted any end-user usability testing of the new checkout experience: so I took a week. We gave early access to 1000 of our most hardcore supporters, and conducted a survey to collect feedback. I analyzed the results and noticed a disturbing pattern in the comments and the log files:
The cart abandonment rate was in the double-digits too high. A pending disaster! So I set to work planning video-recordings for in-person usability tests. I put newbies in front of the new shopping cart, gave them some tasks to accomplish, and set them to work. I didn’t say anything. Just watched them use the cart.
I noticed that during the checkout, people were having a hard time with error feedback in the checkout form. With that data in-hand I made some tiny changes to an open-source project on GitHub (note: recorded on GitHub, not in our own issue tracker).
After a while, I ran the same analysis I did before. The shopping cart abandonment rate was reduced by double digits: A difference worth more than $1 million dollars per month.
That week, I closed one ticket in Jira. One multi-million-dollar ticket.
When you look at tickets wrong, they’re not just meaningless… they’re horribly misleading. Tasks relate to value only in the aggregate, and only viewed independently of “commitments” to the business and marketing teams.
At the micro-level encouraged by scrum, tickets have arbitrary value that differs on a powerlaw curve, but they’re all viewed the same.
What is a powerlaw?
The top 20% of the tickets account for 80% of the company success. Tickets are not created equal, but we treat them like they are.
Tasks are to business value as a finishing nail is to a skyscraper. Conversely, one task might be, “pour the foundation”.
Without that one task, the building could not even exist. Sometimes we assign arbitrary points to tickets, hoping to weight them more realistically, but honestly, that’s like trying to scoop up the ocean in a bucket. It’s a lost cause. Starting to see my point?
Stop wasting your time in pointing meetings.
Management is tangled in a constant struggle to force velocity to match commitments. They think that assigning arbitrary value to tickets is going to make that happen. But it never works. And they’re always frustrated that the dev team consistently over-promises and under-delivers. There’s a constant threat that heads will roll. Sound familiar?
I have news for you: None of those things are agile. Scrum is not agile. Ticket counting is not agile. Velocity measurement is not agile.
What is agile? Your team’s ability to adapt and respond to change is agile.
Go look in the mirror and ask yourself:
“Do I want my team to value the number of tickets closed, or do I want them to value our critical business KPIs?”
The essence of agile is the continual process of improving efficiency.
This is the only thing you really need to know about agile in order to put it to (productive) work in your organization. You should value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
If that doesn’t look familiar, you really need to read this.
I’ll add some more of my favorite dev team values:
Skills over titles
Continuous delivery over deadlines
Support over blame
Collaboration over competition
Why is your team’s agile process so dysfunctional? Chances are you’re losing track of those values, and you’re trying to cram the same old chaotic waterfall into agile trappings like scrum meetings and retrospectives.
Face to face scrum meetings can be counter-productive because they encourage competition over collaboration (employees compare themselves to each other), blame over support, deadlines over continuous delivery (“we have to close x tickets before Friday or we’ll have to work the weekend!”), and a pecking order that infects a team like cancer.
Where in the agile manifesto does it say, “stand up for 15 minutes every day and feel slow-shamed because the coder across from you finished 6 tickets yesterday and you only finished one”?
There’s always the obnoxious over-achiever who comes in early and goes home late every day, and closes 2x as many tickets as the rest of the team. Note: almost always a relatively inexperienced, but super eager and super impressive. Too bad they’re slowing down the rest of the team.
Likewise, there’s always the slow-poke who closes one or two tickets. Odd. They’re a great mentor, and they’re always chiming in on other people’s tickets, helping them get unstuck, teaching, and giving great advice. They should be able to close 10x as many tickets as the rest of us, right? They must simply be terrible at time management. (Hint: They’re not. You just suck at evaluating employees).
Ticket counting and “velocity tracking” are the worst ideas in software development management since waterfall chaos.
Forget Counting Tickets
Forget points. Forget estimates and commitments. Estimates are all worthless lies. Try weekly demos, instead. Get the marketing team to hype the features you finished last month, not the features you think you might be able to finish next month.
Build yourself a good feature toggle and marketing release management system and you can still release features according to a marketing hype schedule, but you’ll be hyping finished features, and your team will never have to burn out on night & weekend crunch times again.
Tip: Your marketing and sales teams should never be allowed to discuss features “in the pipeline”, and your sales & biz dev teams should never be able to commit to a deadline without a really flexible MVP (minimum viable product) plan. When I say flexible, I mean flexible:
e.g., Elon Musk is taking us to Mars. Initial sales MVP: get a balloon into the clouds.
Engineering estimates are usually wrong by orders of magnitude, but nobody wants to face up to that fact and deal in reality.
What Should We Measure?
The only thing that matters in software development is that the users love your software.
Everything else in this list serves that purpose. Remember that. Now bring on the real metrics!
The first 5 of these metrics are all essential business key performance indicators (KPIs) for nearly every app developer.
You’re going to wonder why I’m sharing these with a bunch of developers and telling you that these are the metrics you need to focus on, but bear with me. The remaining metrics will clear that up for you.
1. Revenue
None of the other metrics mean a damn thing if you go broke. If you run out of fuel, it’s game over. You’re done. Pack up your office and go home.
Core Tactics:
- Conversion rate optimization
- Crawlable content
- Sharable content
- Page load & perf
Optimize to keep the lights on.
2. Monthly Active Users (MAU)
Do you have any users? Do you have more than last month? If you’re a venture funded startup, you’d better pray you’re growing fast enough. In the beginning you can double month over month if you work hard. Of course, all hockey-stick curves are really S curves in disguise, but chances are good you have plenty of room left to grow.
Core Tactics:
- Page load & perf
- Sharable content
- TDD & code review
Optimize for growth.
3. Net Promoter Score (NPS)
Remember when I said if you run out of fuel, it’s game over? I tricked you, didn’t I? You thought I was talking about money.
Money isn’t your fuel. Fans are your fuel.
Fans are the key to unlocking more money. More sharing. More growth. More joy. More magic.
Core Tactics:
- TDD & code review
- Page load & perf
- Collaboration with support & QA staff
Optimize to turn users into evangelists.
4. Viral Factor
Also known as k-factor or viral quotient. If you’re not measuring this, start right now:
k = i * c
i = number of invites (e.g., shares) per user
c = conversion rate per share
- k = 1 is steady. No growth. No decline.
- k > 1 means exponential growth.
- k < 1 means exponential decline.
You should have a giant screen in the middle of the office with a k-factor gauge in bright red for <1, green for >1, overlaid on your 3-month MAU growth chart.
Core Tactics:
- Sharable content
- Integrate sharing into core product
- Conversion rate optimization
- Page load & perf
Optimize for sharing and new visitor conversion.
5. Support tickets
Nothing says “this garbage is broken” like an email to customer support. When was the last time somebody contacted support just to tell you how cool you are?
Support tickets are your canary in the coal mine. When somebody says something is broken, don’t think, “it works for me!”. Even if it’s user error, it’s not user error. Chances are there’s a flaw in the design and 1,000 other people are bothered by it too. 1,000 other people for every one person who cares enough to write you and complain. 1,000 people who’d rather hit the back button than waste their time on your app for one more second.
Ideally, you should aim for zero support tickets. You’ll never reach that metric (if you’re lucky), but you should consider every support ticket to be a bug report. Start categorizing the common ones. Count them and use them to prioritize fixes.
I’m not saying the customer is always right. Sometimes customers don’t know what they want until you give it to them. I am saying that if it’s in your inbox, you’re doing something wrong.
Core Tactics:
- TDD & code review
- CI/CD
- Feature toggle & rollout
- Periodic bug burndown hackathons
- Collaboration with support & QA staff
Optimize for problem-free customer experience.
Key Engineering Focus Metrics
As promised. The keys to unlocking the mysteries of the business KPIs. As it turns out, you can move all of the above needles a lot with two levers:
6. Bug Count
Here’s a shock. Some ticket counts are good for something. Be careful to categorize all the bug tickets as bugs, and then you can see a bug count.
All software has bugs, but not all bugs need fixing. If a bug appears only on an ancient phone, and that phone is only used by one user of your software, and that user isn’t even a paying customer, do you need to fix that bug? Probably not. Close it and move on. Prioritize the bugs that are hurting your users the most.
Get busy and squash them.
Core Tactics:
- TDD & code review
- Periodic bug burndown hackathons
- Collaboration with support & QA staff
Optimize for a bug-free experience.
7. Performance
I’m cheating a little this time. This one is going to contain 3 more critical metrics:
- Load time: The time it takes for your app to be usable after the user clicks the icon or hits your URL. Aim for <1 second. Beyond that, you lose users. Every ms you can trim off load time comes with measurable benefits to every metric above.
- Response time: The time from user action (like a click) to a visible response in the application. Aim for <100ms. Any more than that feels like lag.
- Animation time: The maximum time it takes to draw one animation frame. Aim for 10ms. Any more than 16 ms will cause noticeable jank, and may even make the user feel a bit queasy. You’ll need a little breathing room for this one. Keep it under 10ms.
Load time is by far the most important mini metric in this list. It will move the business KPI needles like nothing else I’ve mentioned. But response time and smooth animations cause a magical side-effect. Users are happier after using your app. You see, every little janky glitch, every little delayed response feels jarring to users on an almost subconscious level.
Give the same user the same app with performance issues fixed, and they report much higher satisfaction ratings, even if they can’t put their finger on why.
Our little secret.
Core Tactics:
- Periodic performance hackathons
- In-depth performance audits
- 10ms, 100ms, 1000ms, repeat
Optimize for a jank-free experience.
There’s A Lot More To It
Of course, this little rant can’t go into great depth on how developers can directly manipulate viral factor and MAU numbers, but I’ll leave you with a hint: You can. And when those numbers are staring you in the face every day, and you know that it’s your job to move them — not management’s job, not marketing’s job — your job, I’m sure you’ll come up with some creative ideas to make it happen.
If your manager thinks you have better things to do, send them this link.
Now go out there and move the needle on some metrics that actually matter.
Eric Elliott is the author of “Programming JavaScript Applications” (O’Reilly), and “Learn Universal JavaScript App Development with Node & React”. He has contributed to software experiences for Adobe Systems, Zumba Fitness, The Wall Street Journal, ESPN, BBC, and top recording artists including Usher, Frank Ocean, Metallica, and many more.
He spends most of his time in the San Francisco Bay Area with the most beautiful woman in the world.
