Caught in the Middle: Life as an Analyst in 2022
Should analysts be technical and build data pipelines or should they focus on answering business questions? When I speak to data leaders, data roles are converging around data engineer, analytics engineer, data analyst, data scientist and machine learning engineer.
One role still means different things to different people. The analyst.
In some companies analysts work end-end, build data models in dbt, write tests and monitor jobs in Airflow.
In other companies, analysts solely focus on business problems, dashboards and insights while analytics engineers are responsible for data models.
Life as an analyst in 2022
We expect much more from analysts today than we did just five years ago. As companies become more data driven, we rely on analysts to tell us what to do (sometimes too much). At the same time, data is more complex and comes from dozens of systems which has led to data teams starting to adopt engineering best practices such as testing, version control and CI/CD.
If you’re an analyst this means that you’re often caught in the middle. You’re expected to both have a good understanding of how the business works as well as a deep technical understanding.
In other words, the data analyst role has never been harder.
Analyst in 2015
- You worked in Excel, used a BI tool and perhaps wrote some SQL that you learned on www.w3schools.com and saved the scripts locally on your desktop
- You worked on clearly defined OLAP cubes that were made by the BI team and if you wanted to add new columns you’d have to wait for weeks for the BI team to add them
- You spent a lot of time updating spreadsheets and PDF reports for the weekly KPI review meeting. This was tedious but also helped you develop an intuition for the numbers
Analyst in 2022
- Your company uses a modern data stack and you’re pulled into everything from writing tests in dbt to debugging Airflow pipelines. At the same time you’re expected to stay close to business problems
- Your company uses ELT instead of ETL and all the data you’d ever dream of is easily available. But with so much data you’re struggling to stay on top of the pipelines and data quality has become a real issue. Metrics are defined all over the place and you’re drowning in Slack alerts from dbt and Airflow and don’t know which ones to pay attention to
- People keep asking you to read an article about a new concept called the Data Mesh but you don’t really get what it’s about
Being an analyst in 2022 is more exciting but it’s far from perfect.
We’re risking making the data industry look too daunting to new analysts just starting out. In my experience some of the best analysts often come from unexpected places. New data tools should reduce the barriers to enter the data industry, not increase them
Should analysts own data pipelines
Maybe. If it means that they can unblock themselves and work faster, then yes. If it means that they’re removed a step further from understanding the business and spend most of their time on technical work, then no.
Reasons for having analysts own data pipelines
- They’re not reliant on analytics engineers to unblock them to get the data they need
- They up-skill on technical tools and develop an engineering-like way of thinking about data
- They’re more likely to take responsibility for the quality of the data they work with
- They can have a seat at the engineering table and be part of helping shape how new products should be designed from a data lens
Reasons for having analysts solely focus on insights, analysis and dashboards
- They have more time to do investigative Sherlock Holmes like work
- Analytics engineers and data engineers can do what they do best and make sure data pipelines are high quality and consistent
- Some of the best analysts I’ve seen have self-taught data skills to solve specific problems they faced in a sales or operations role. If we make the barrier to enter the data space too high with complex tools and workflows this becomes increasingly unlikely
Wish list
We’re at a crossroad and while there’s been some exciting developments in the data tooling space I have three items on my data Christmas wish list
- Tools to augment the data workflow. With the modern data stack, data people have to spend too long debugging what went wrong when a dbt test fails or when trying to asses the downstream impact of changing a column in a data model
- One place to define metrics. dbt announced metrics at Coalesce last week. This will be the future and having metrics defined in one place will make it much easier to keep one set of metrics that’s used throughout the company
- Tools that get out of the way. Data tooling should be so good that analysts spend 90% of their time on insights instead of 90% of their time on debugging data models and checking errors in dbt or Airflow
Watching talks such as last week’s Building an Open Source Data Stack with Katie from Lightdash makes me hopeful that many tools are heading in this direction.
