Wait…how come your number is different than mine?
How Atlassian built a tool to share metric queries across its analyst teams
Analytics at Atlassian has rapidly expanded over the past few years. We now have over 400 data scientists, data engineers, platform engineers, and machine learning engineers working on a broad array of problems. The technology we use has also undergone dramatic changes — it is amazing to fathom that at one point, analysts relied solely on a postgres database running on a single server!
The postgres database of course ended up not scaling well as the amount of data ballooned. Fortunately, there are many solutions out there to handle data at scale. On the other hand, we were learning some processes also face scaling issues when your team grows significantly. It’s easy to share information when an analytics group only includes 10 people; important changes to a table, logic, or system are easily communicated. But when you have 150 people, how can we make sure everyone is on the same page? This became the more difficult problem to tackle.
At first, the loose process was just a nuisance but it grew into an enterprise problem when analysts started reporting different numbers on key internal metrics, such as monthly active users. Differences arose in the chosen tables or the filtering criteria causing confusion among teams and stakeholders. Efforts to centralize the queries into repositories only slightly improved the situation, but still people did not check the sources regularly enough, or the queries became stale because maintainers forgot to update them. And the problem was only escalating as more and more people were onboarded and it became more challenging to change the process. There had to be a better way…
At Atlassian, we adhere very strongly to our company values and encourage all employees to embrace these values. And one team member Amit Tiroshi ruminated over one particular core value when he was dealing with the metrics chaos: Be the change you seek. He didn’t know exactly how to solve the metrics decentralization problem but decided to give it a shot.
Amit approached the metrics problem by borrowing a strategy he learned from working on an Atlassian product team: customer interviews. He setup time with analysts to figure out why previous solutions had failed. And he found a couple of key problems and requirements:
- Navigating to a specific page or repo to retrieve the latest queries was too cumbersome. And if the pages are not visited frequently, it becomes out-of-sight, out-of-mind and people forget to maintain it. A solution had to be convenient.
- Each analyst had their own preferred platform. Some people used Databricks, others used Redash, others connected locally. The solution had to be platform agnostic.
He began tinkering with some ideas and sourced a few open source projects that he could leverage. Fortunately, Atlassian gives its employees an encouraging environment to do exactly this type of brainstorming and prototyping through its quarterly ShipIt Days. It’s a 24-hour competition to allow teams to ideate, build, and share innovative technical and non-technical projects. It was a perfect opportunity to build a v0 solution, and thus the Atlas Data Kit was born.
Amit’s Atlas Data Kit is an OSX application that helps you find key metrics and copies the query to your clipboard. The actual queries are source-controlled in a Bitbucket repository and synced automatically. No more mistakes, perusing through documentation, or bugging others to get the logic you need. It’s all there at the top of your toolbar.
The next step was enhancing the application for a broader audience. Fortunately, managers recognized its value and invested the resources to scale it out. The first obstacle was navigating security so that employees did not have to do any special configuration. We wanted analysts to be able to install the application and sync to the source repo without exposing it publicly. Our engineering team helped implement a solution to do that successfully. Additional enhancements included syncing the query metadata to Atlassian’s internal metric documentation, so everything is always up-to-date. We then formalized the peer review process, so we could ensure quality without introducing a bottleneck. Finally, we’ve been working on promoting the tool internally to achieve greater adoption and contributions, and adding Atlas Data Kit demos to analytics onboarding and brown bag sessions to train our newest hires.
For those Atlassians who have adopted Atlas Data Kit, it has helped remove many communication challenges when updating a query definition. It has ensured consistency and reduced errors. The repository also makes it easy to find the recent query contributor, since that information is captured in the git commit history. The peer review process enforces accurate and well-written queries. And Atlas Data Kit uses an auto-sync to make sure we all are using the latest versions. Our teams can now spend more time solving new problems and less time chasing query definitions.
Analytics at Atlassian is constantly evolving. Existing teams are expanding, and new ones are being formed. We still face challenges, but we have a strong culture of taking the initiative to find or build a better way. Atlas Data Kit is one terrific solution to our challenge around metrics sharing. We look forward to publishing more stories about Data and Analytics at Atlassian in future blogs. Thanks for reading!
Thanks to: Amit Tiroshi, Reed Johnson, Olli Fleur, Hideyoshi Cheong, and Tristan Frizza for their work on Atlas Data Kit
