What do Analytics Engineers Actually Do?

The move towards cloud data warehouses, the evolution of self service BI tools, and introduction of data ingestion tools has led to some big shifts in data tooling. These shifts have meant that the roles and responsibilities that existed in traditional data teams are also changing. Enter the Analytics Engineer.

At Dataform we know that a lot of our users are doing the work of analytics engineers but aren’t necessarily calling themselves this yet. So what does it actually mean to be an analytics engineer? What skills do you need? And how does the role differ from traditional data roles? We sat down with several Analytics Engineers to discuss what the role means to them.

Where does an Analytics Engineer sit in the data team?

Before we begin, it’s useful to define what roles exist in a traditional data team and where an Analytics Engineer sits amongst these:

• Data Engineers: manage core data infrastructure, ensuring data is available and accessible across the organisation.
• Data Analysts: partner with business stakeholders to answer questions with data, build dashboards and reporting, and carry out exploratory analysis.
• Data Scientists: use statistics and machine learning to extract value from data: solving optimization problems, building prediction models, running A/B experiments and more.

The shift to an ELT approach means that data now lands in the warehouse before it has been transformed. This presents an opportunity for super technical analysts who both know the business really well and have the technical skills to model the raw data into clean, well defined datasets to step up (analytics engineers). If you wanted this kind of analyst when using the ETL paradigm they would need to be a software engineer and a data analyst — and that would be a lot harder to find. Finding a technical analyst is a lot more achievable.

The Analytics Engineer acts as the bridge between data engineers and data analysts. It is their job to build well tested, up to date and documented datasets that the rest of the company can use to answer their own questions. They are technical enough to easily be able to apply software engineering best practices like version control and CI/CD but also need to be able to communicate effectively with stakeholders to teach them about these things. Saira Barles, Analytics Engineer @ Hubspot describes the differences between the 3 roles using the analogy of a cupboard:

“Data engineers build the cupboard, they gather together the wood and the tools and put it together. The Analytics Engineers open the cupboard and start putting in the plates, mugs, bowls, and arrange them in a certain order. This could be arranging them into particular colours, shapes or sizes. Data analysts then go into the cupboard and they know where everything lives as it is arranged nicely. They can then grab the small blue mug they were looking for and go make a cup of tea!”

What are the responsibilities of an analytics engineer?

Although the exact responsibilities of the role differ from company to company, fundamentally, it is the job of the Analytics Engineer to build data products that enable stakeholders to do their jobs effectively and answer their own questions. Mitchell Silverman, Analytics Engineer at Spotify breaks down how he sees his responsibilities:

“50–60% of the job is building out datasets and setting up monitoring, testing and automation around those. 20% is more of a consultant role, so how to enable analysts and data scientists to do their jobs better e.g. Cleaning up scripts/queries they’ve written and showing them how they can improve. The final 15–20% is miscellaneous. Analytics engineers come from all different backgrounds so it kind of depends where your skill set lies”

Sometimes analytics engineers are responsible for ingesting data into the warehouse, however more often it is the data engineer who does this (especially at larger companies). The analytics engineer then picks it up from there and makes the data usable. They are in charge of modelling raw data into datasets that enable analytics across the company. These datasets act as a centralised source of truth.

A big part of the job is making sure that these datasets are maintained. The datasets need to be up to date, well tested and documented . The Analytics Engineer should be the first person to be notified if something in the pipeline breaks so they can fix it and should be empowered to own that whole workflow to ensure the data models are running reliably.

They are also responsible for enabling data analysts and data scientists to be more effective. Whether that’s sharing tips for writing better SQL, updating a dataset to contain a new metric or dimension, or coaching them on how to adopt software engineering best practices.

How do you become an Analytics Engineer?

We spoke to various different Analytics Engineers from a variety of backgrounds and found some common skills and experience that they considered to be important:

• Comfortable adopting software engineering best practices like version control and using Git. Teaching others how to adopt these best practices.
• Be a SQL whiz! They know how to write SQL which is easy to understand, simple to troubleshoot and that is highly performant.
• Knowledge of another programming language e.g. Python is a plus.
• Experience working as part of a data team; preferably as either a data analyst or data engineer.
• Able to build and maintain multi-functional relationships with various teams across the business.

People can go directly into analytics engineering from a variety of backgrounds. Mitchell from Spotify says that on his team he typically works with people who “used to be analysts or data scientists who for whatever reason ended up getting more into software engineering and doing some backend work, and over time have built up the skill set.”

Kadu, Analytics Engineer at Liv Up, describes most people who go into analytics engineering as coming from a few different backgrounds. Either they come from a software engineering background who focus more on data pipelines — performance and reliability above all — or they come from a data science background and focus more on getting people the correct data they need, even if pipelines end up breaking sometimes.

One common thread was that everyone we spoke to agreed that having a healthy mix of technical and stakeholder management skills was crucial. Saira at Hubspot describes why she loves the role “I love data. I love SQL. I love transforming data and reducing 20 lines of code into 5! But I also want to solve problems for the various stakeholders at my company and make their lives easier”

What tools do Analytics Engineers use?

This varied slightly from company to company, but all analytics engineers we spoke to used:

• A cloud data warehouse for storing data (Snowflake, Bigquery, Redshift)
• Fivetran or Stitch for data extraction and loading
• Dataform or another data modelling tool for data transformation
• A modern BI tool (Looker, Mode or Chartio)

Traditionally the “first data hires” at a company were probably a data engineer and a data analyst. The data engineer would be tasked with setting up infrastructure, getting data flowing into the warehouse and ready for analytics. The data analyst would then build reports and charts on this data that they could share with the business.

The introduction of the tools mentioned above means that the “first data hire” can now own that whole workflow. Tools like Fivetran and Stitch allow people who may not have the skillset to do complex data ingestion tasks to call themselves “data engineers”. Looker has made it easy for stakeholders to answer their own questions and create their own charts.

Traditionally analysts would build a pipeline but wouldn’t know if it had broken until an end user spotted an error in their dashboards and reported it. Nowadays that isn’t acceptable and data teams need to know immediately if something breaks. Mitchell at Spotify says that the biggest value-add that data transformation tools like Dataform bring is “monitoring and testing”.

Dataform is championing the analytics engineer’s mission. Anyone who is comfortable writing SQL can own the transformation piece of the pipeline. Kadu at Livup says that they have “15 analysts all managing their own ELT pipelines using Dataform”.

How will the role evolve?

Nobody really knows what the future of this role is. Most companies don’t have a big analytics engineering department and there are no Head of Analytics Engineering roles (yet), so there’s no set career path. Opportunities to move sideways into Data Engineering, Analytics or Data Science, possibly in leadership roles, may present themselves. A path into Product Management could also be possible.

In order for Analytics Engineering to become a recognized field the first step is for more companies to acknowledge it. Mitchell says that at Spotify there are “only 10–15 Analytics Engineers amongst 100s of data analysts and scientists”. It is their job to show to the rest of the company the value they bring and why Spotify should invest in setting up a talent pipeline for this field.

Kadu from Liv Up explains that in Brazil most companies expect Data Engineers to perform both roles, even in larger companies. “Many people are being called Data Engineers when they are actually doing the job of Analytics Engineers”. Each role requires a very different skillset.

Therefore, in order to build a cohesive cupboard with your plates and mugs arranged correctly, it’s important that companies understand the differences between the two roles and hire accordingly. Despite this, more job postings for analytics engineers are being posted everyday, communities around this role are rapidly growing, and tools which help their workflows are evolving faster than ever before.

It’s an exciting time to be an Analytics Engineer!

Originally published at https://dataform.co.