Three years of Analytics Engineering at Slido: How a team of 3 manages to support a company of 260
A lot can change in three years. Three years ago Analytics Engineering was a niche role, Slido was not part of Cisco, and our masks were securely on our faces instead of being stuffed, forgotten, in pockets, drawers, and in the glove box of our cars. In three years, so much has changed about the way we work & live.
But some things haven’t changed. Three years ago a core team of just 3 Analytics Engineers scaled Slido Data from ad-hoc analyses & Excel to streamlined CI/CD pipelines and Software Engineering best practices. The team is still just 3 today. In this article, we’ll take a look back at what the team did to make that possible.
Full disclosure — I joined Slido Analytics Engineering in February 2022, to replace another AE who was leaving for maternity, so despite not being part of the original 3, I have been around for half of the fun. This is also why the narrative voice moves from ‘they’ to ‘we’ about halfway through the article 😬
Let’s set the scene — it’s early 2021 and our analytics team of three are in a Google Sheets and custom query state of existence. The principal aim of the team is to answer burning questions, making a reasonable call between speed and precision. Little to no formal attention is paid to the methodology of how the data is obtained. Code is written and rewritten locally, CSVs are monotonously imported, and tables are endlessly pivoted. The analytics team is essentially a service function, answering incoming questions like ‘How much money did we make last week?’, and ‘How many Slido events were run last month?’. There were no dashboards back then and our manually updated spreadsheet-based reporting was very error-prone, causing trust in data to be very low.
Many of us ‘Data People’ will squirm at the thought of such monkey work. We didn’t spend hours mastering our craft just to run a SQL query and copy-paste a few hundred rows of data into a Google Sheet. Google Sheets doesn’t even have good keyboard shortcuts, for goodness sake! However, it didn’t feel (much) like that for the then-Data team. They were central to a larger team of innovators — bringing a product that was forged in a start-up competition to the European market. Slido’s mission was and still is, to transform how meetings are run, facilitating inclusive conversations at meetings and events all across the world. Being mission-driven was the name of the game, and a few hastily written SQL blocks and pivot tables were not going to stop that!
Of course, our Analysts also knew that being the first Slido Data People, they would have an opportunity to shape how the data organisation would evolve. Amongst the frenzy of pulling numbers and fixing Vlookups, the Analytics Team made space to consider their options. The aforementioned frenzy encouraged the opinion that what the team needed was oversight & standardisation. The team looked for and found an open-source solution (dbt) that allowed code should be reviewed, stored, and reused, and workflows aligned. The introduction of dbt corrected much of the duplication of work, as well as instated code control systems. Drawing heavily from Software Engineering practices, dbt enables analysts and Analytics Engineers to produce peer-reviewed code, through unified and optimised workflows, whilst creating and managing documentation within the same interface. Datasets & metrics definitions were standardised, and basic dashboards and internal documentation (including knowledge articles) were created. The move was impactful — the analysts became more confident in their work knowing that checks and controls were in place for the code. The rest of the company became more confident in the data knowing that there was a single source of truth for reporting, metrics & documentation, and a knowledge hub for when they got stuck. With this, a data culture began to form. Decisions were not based on gut feeling because of the centralised resources & dashboards, & the feeling of self-sufficiency and trust grew.
The roll-out and ramp-up of dbt occupied a large part of 2021, and as 2022 rolled in, the team’s focus pivoted to putting the new, shiny, version-controlled, peer-reviewed, & documented dbt project to good use.
We looked for opportunities to create impactful wins for the business to show off the newfound order in the data, as well as present a team that was no longer only focused on reactive service tasks. Data ‘products’, serving a functional purpose to fix a problem or provide a service, would be created in closed development loops to avoid run-away timelines. Their impact would be clear and measurable, and they’d serve as a great example of how the data team was becoming a collaboration partner instead of a support team.
The first products were two Python libraries; dbt-coverage, a single CLI tool which checks your dbt project for missing documentation and tests, and dbt-superset-lineage, a package for extracting dashboards from Apache Superset to dbt docs as exposures. The packages eliminate the need for mountains of manual work, removing the burden of monotonous tasks from the analytics workflow and allowing the same resources to be better allocated. The packages were a huge benefit for the team themselves, but also for others outside the organisation, since dbt is open source (aligning nicely with Slido’s vision of being fully collaborative — sharing is caring!).
The biggest outward-facing data product was DataPi (and the topic of last year’s presentation at Coalesce — dbt’s own Analytics Engineering conference). DataPi is Slido’s proprietary data API which enables clients to automate the access, retrieval, and storage of their data. Many of the big enterprise clients that the Sales team were after are legally obligated to extract and store Slido usage data. To be able to close deals with these clients, we needed a solution to automate the extraction. Enter DataPi. DataPi was fully built, managed, and documented in dbt, parallel to the internal dbt project. This was the first time that the Analytics Engineers had been among the primary responsibles for the release of a client-facing data product — which we think is pretty cool. Most importantly, DataPI enabled the sales team to land deals with big clients in a way that they’d not done before — and the Analytics Engineers had a massive hand in making that possible
There was also time to focus on a data education strategy by building helpful products for our colleagues in other teams. We created Slack dat-bots to send automated, data enriched, messages directly to relevant Slack channels — the Sales team gets sales pipeline metrics, customer success teams get NPS data, and the whole company gets weekly company KPIs. These bite-sized data nuggets are much more easily digested than a whole dashboard and are a more engaging way to get people closer to the data. Learning resources were also productionised, creating two parallel approaches — hands-off and hands-on. Hands-off, we created Slido-customised teaching assets (in hard and soft skills), and a data-skills matrix to support people navigating through their Learning and Development journey. Hands-on, we ran training sessions and created a ‘Data Champions’ programme for people who want to get closer to data in their field of expertise. Non-data teams became more self-sufficient through these internal education products and as a result, gained the vital confidence needed to answer ever more complex questions that arise in a growing company.
In 2023 our aim has been to refocus on collaboration with our business colleagues and impact business metrics more than ever before. We’re working to increase the speed to impact by using learnings from deeper working relationships with business teams to anticipate questions before they are asked. We’re creating prediction and alerting tools to make our impact more active and less reactive, and we plan to implement the semantic layer to enable greater flexibility and customisation in our reporting tools.
It may be easy to imagine that with all this progress the team has been growing over the past couple of years. Actually, the core team started off as 2 people, and increased to 3 soon after its inception, but since then has remained the same size. At the same time, the entire Slido team has blossomed from 150 in January 2021 to over 250 people today. Three people have managed to support a 66% growth in colleagues over a three-year period. The principal concept from which we have gained so much is the scalability of this approach. We are all in absolute agreement that this would not have been remotely possible without the decision we took years ago to create oversight & standardise. The software engineering mindset, as well as the dbt environment and tooling, have been crucial in enabling 3 Analytics Engineers to scale data in Slido’s growing team in only three years. Our impact has been considerable. Imagine what we could do with 2 extra people…!
