What is the difference between a BI Developer and a Data Engineer?
Comparison between collecting and organising the ingredients vs garnishing the plate before serving it!
The ever increasing complexity of data in our digitalised world means that companies are investing substantial parts of their resources in talent that can navigate these pools of information. For someone who is interested in a career in data, the amount of opportunities and various roles that are out there can therefore often seem overwhelming.
As part of our blog post series on the different data roles at Gousto, we will take a closer look at BI Developers and Data Engineers today. We have spoken to Elena Martina (Data Engineer) and Sarah Lier (Senior BI Developer) to learn more about what they do. They are part of the Data Platform team at Gousto (nicknamed “Passionfruits”) which consists of Data Engineers, Analytics Engineers, BI Developers and a Business Analyst.
What are BI Developers and Data Engineers?
Elena: An engineer’s role is to design, build, and maintain the systems or infrastructure to solve problems. As such, Data Engineers are the designers, creators and maintainers of optimised systems for collecting, moving, storing, and organising data at scale. Our main goal is to create a single source of truth (in Gousto’s case, in our Data Lake) for the users of our data and provide them with highly reliable, usable data in near real time.
In my team, Data Engineers take care of getting the data from different sources, loading it into our Data Lake in an unrefined “Raw” layer, then processing it into a second “Cooked” layer (we’ve got food puns everywhere!). In the Cooked layer, data is stored in Delta and it’s cleaned, partitioned, and optimised for querying. Transformations and modelling of this data happen afterwards from “Cooked” to “Served”, thanks to our Analytics Engineers and BI Developers.
Imagine a chef getting their groceries, bringing them back to the restaurant and organising them in the most efficient way for them to be ready and available when the restaurant opens and the orders start. They will make sure that all the ingredients are there, and they’re all in perfect condition (no incorrect spices or damaged veggies!), as well as arranging them in the best way to save storage space and make them easily reachable when needed. That’s kind of what we do… but with data!
Sarah: A BI Developer — in a quite literal sense — develops business intelligence through various analytical and visualisation tools, for example Power BI, Tableau, Looker or Sisense. The resulting reports enable stakeholders and the wider business to make crucial decisions based on historical data, present trends, and forecasting.
If the chef in Elena’s example makes all the ingredients available, then a BI Developer would take over from there. They would mix the ingredients, make a delicious pasta bake, and then serve it to our stakeholders
Does your role definition vary from company to company?
Sarah: Yes, because the data journey at a company depends on the company size and the maturity of its data department. At companies with less complex data sets or a more rudimentary data infrastructure, the BI Developer often does the job of a Data or Analytics Engineer, too. This means that you will also ingest, transform, and model the data before you plug it into an analytics tool. The more advanced the data team of a company is, the more structured your responsibilities become. However, this does not mean that BI Developers and Data Engineers are completely separate but we will get to that later.
Elena: Exactly what Sarah mentioned! The role of a Data Engineer in a company is very dependent on the company size and the maturity of its data team. The role and required skills also vary by company, as it heavily depends on the scale of data that’s being ingested, the types of data sources (different skills will be required for different use cases) and the latency requirements of their data users (does your company need real-time, streaming and/or batch data?)
How did you become a BI Developer and Data Engineer?
Sarah: I have a background in humanities and wanted to pursue a career in academia which did not happen in the end for various reasons. It was therefore time for a career change, so I thought about what else I like. I remembered that I always enjoyed my Computer Science, Maths, and Physics classes at school. At the same time, I observed the growing importance of data to companies. I subsequently joined a market research company as an Analyst where I got to use Power BI for the first time. From there, I found my way into BI.
Elena: Funnily enough I have an early background in humanities too, however I had always been more the analytical type and I never quite fit in. Participating in a project for women in Tech inspired me to pursue a career in Tech and follow the path of the amazing women I met there. I decided to come to London to study Computer Science at university; during my third year I got closer to Data, and I realised it was the space I wanted to work in.
I was initially looking for Machine Learning Engineer roles; however, since landing my Data Engineer role at Gousto, it’s been such an interesting, rewarding and fun role, that I haven’t looked back to Data Science.
You’re both part of the Data Platform Passionfruits team. What does your typical day look like in Passionfruits?
Elena: You’ll usually find me starting my day around 9am, sipping a warm cup of moka pot coffee in front of Slack while answering messages, planning my day and maybe reviewing a few PRs. Twice a week, I join an optional team-wide morning “coffee and chit chat”, which is a good way to ease into the day and catch up with everyone.
Standup with my feature team is at 10am, which is when my team’s core working hours actually start. It is usually followed by a refinement, sometimes a retro, or people splitting up in a pairing session to continue a discussion, task or issue brought up during standup.
After lunch is when I usually have time to focus on the projects I’m working on, either on my own or pairing with a colleague. This afternoon, I’m continuing to work on an ML Ops project, which will enable Data Scientists to easily deploy their Machine Learning models to Databricks using dbx.
Sarah: Our typical day consists of team tasks, individual tasks and then what I call ‘Gousto extra-curriculars’. Our team tasks often come in the form of meetings, for example retros where we talk about how we are progressing against our OKRs and tech slots where we present what we have been working on.
On an individual level, I have a lot of technical work which is about the development of reports for our stakeholders. However, as I am at a Senior level, I then also work on the bigger picture for our team and the business. I own OKRs, think about larger problems, and come up with solutions.
Our ‘Gousto extra-curriculars’ consist of voluntary activities, for example contributing to our TechBlog. Elena and I also co-organise get-togethers and are involved with the Women in Tech at Gousto working group.
What skills are most crucial for your job?
Sarah: It’s a mix of technical and interpersonal skills. In terms of more fundamental technical skills, you need to know what entity-relationship models are and how relational databases work. Once you have a good grasp of that, you can dive into the more specific technical skills, i.e. the actual business intelligence tools. As they are all fairly similar, you can usually be an expert in one tool, for example Power BI or Looker, but then still join a company that uses Tableau. Many tools use SQL, though Power BI runs on DAX and Sisense includes elements of JavaScript. Again, if you understand how one language works, you can easily pick up on another one.
Then there are the interpersonal skills. As a BI Developer, you are often the face of the Data team and spend a lot of time liaising with stakeholders. Your stakeholders are not always technical, so you need to be patient and never assume that they know what you know. I like the comparison to declaring variables when you code. The computer may not know that ‘age’ is an integer, so you need to declare it. Another way of approaching the role of a BI Developer is the service-angle. I think of my work as providing a service to my colleagues which means that I am very invested in their needs.
Elena: In the two years I’ve been at Gousto, the most useful skills have been the ability to learn fast and get comfortable quickly with new tools and technologies, while keeping up with what’s new in the Data space. Being able to pair and collaborate with other engineers, communicate with technical/non-technical people, problem solving, and the love (or laziness?) for automating everything you do.
What tools do you use for your job?
Elena: Gousto is an AWS house, so we use a lot of the services it offers. To integrate with and collect data from our sources into our Raw layer, we use AWS lambdas (for APIs or message buses) or DMS (for databases). AWS S3 is where we store our data, and we use AWS Glue as our data catalog.
Most of our Data Platform relies on Databricks running on AWS: we use it as the orchestration tool for our Spark Streaming jobs, and it’s also used by the entire Data team to query our Data Lake through its notebooks or Databricks SQL. Our Databricks jobs are deployed through Databricks Labs’ DBX. As a Data Platform team, we also own the whole Databricks infrastructure and setup, which we manage through AWS CDK (Cloud Development Kit).
CircleCi is the tool used across Gousto for deployments, and Github actions for automations. Last but not least, Miro is a whiteboarding tool for architecture design and collaboration!
Sarah: The Analytics tool we use at Gousto is Sisense. My background is in Power BI and I had dabbled in Tableau before, so Sisense and JavaScript were new to me when I joined Gousto. I also use Databricks to test our data before I plug it into Sisense or to model raw data tables that should not sit in a BI tool. Moreover, I am currently learning dbt which goes into Elena’s world but which is super interesting and helpful if you want to make changes to a data model without getting an Analytics or Data Engineer involved.
Who are your key stakeholders?
Sarah: My key stakeholders are a mix of technical and non-technical people. On the more technical side, I usually work with Data Analysts, whereas stakeholders from outside the Data team can be anyone from Logistics, Operations, Finance, and so on. It’s quite exciting because it means that I get to know all kinds of different people and departments at Gousto.
Elena: My stakeholders can be any team in the business that uses our data or needs data from a new source ingested in our Data Lake, from marketing to software engineers to data analysts. I also closely support Data Scientists, which I help with queries, model deployments and new data or feature requests.
At times we are our own stakeholders, as I often work on platform improvements and adding new tooling to make my team’s life easier, while adding value and functionality to our Data Platform.
How do BI Developers and Data Engineers interact with each other?
Sarah: Within a Data Platform team, Data Engineers are essentially the ‘back end’ whereas BI Developers are the ‘front end’. Then there are Analytics Engineers who sit in the middle. Data Engineers essentially ingest or integrate data from different sources to consolidate them into one view, usually a data warehouse or lake. The Analytics Engineers then transform and model the data so it is ready for use. BI Developers then take the data and analyse it through BI tools.
At Gousto, all three roles are part of one team (Data Platform) which means that we work together very closely. For example, as a BI Developer I would approach a Data Engineer and ask ‘Our stakeholders need this type of data. Is this available? If not, is it possible to make it available?’
It is important to work together so we can maintain a clean data pipeline and develop a fully integrated data stack. Our end goal is accurate and healthy data that travels smoothly from its starting point to the finishing line.
Elena: I couldn’t have said it better myself!
What advice would you give to someone coming into your profession?
Elena: In terms of technical skills, I would recommend getting some Cloud experience, learning python or Scala, Spark, SQL, and understanding of Cluster Computing and IaC (Infrastructure as Code). In general, don’t be scared or conditioned by your background or skills. Data Engineering is often highly dependent on the data and the systems a company has already in place. No matter how many Data Engineering courses you take beforehand, you will learn a lot of it on the job. So when you’re applying for a role, don’t worry if you don’t “tick all the boxes”. Keep up to date with the latest technologies in the space, choose which ones you want to learn, and apply for the job where you can use them!
Sarah: Just because you did not study a science — or data-related subject at university, it does not mean that you cannot become a BI Developer. There are plenty of free online resources that you can use to obtain training, for example YouTube or edX. If there is an opportunity to ‘buddy-up’ with someone from the data team at your company and get some hands-on training, go for it!
Check out more stories from Gousto and make sure to follow us here, so you catch our next article. While you’re waiting for that you can check out this post that looks back on everything the tech team did in 2021!
