Three roles in Data & Insight at Oda — Part 2: The Data Engineer
What does a Data Engineer do at Oda? Which tools do they use, and how does their daily work look like?
In the initial article of this blog series, Nina Walberg introduced the three roles in Data & Insight in Oda: Data Analyst, Data Scientist, and Data Engineer. She gave a general description of how the roles are defined and what sets them apart. To give you a detailed overview of these roles, we have interviewed three people working at Oda. In this interview, Hanna Heggheim Lee will give you a better understanding of her work as a Data Engineer at Oda.
As of November 2021, our Data Platform team consists of 9 people, most of us are Data Engineers. The mission of the Data Platform team is to empower Data Analysts and Data Scientists to efficiently create value from data. Part of this is ensuring that they have the tools, support, and guidelines that they need to be productive. As you can probably guess, that task has many different components. Some of us work with setting up the necessary infrastructure for our Data & Insight platform, while others work more with data modeling and data pipelines, or the machine learning platform.
What is your background and what attracted you to Oda?
I have a Master of Science degree in nanotechnology from NTNU, with specialization in nanoelectronics. During my first semester, one of the courses I had was introduction to programming. I had never written a line of code, but quickly realized that this was almost as fun as solving equations. After graduating I worked a couple of years in consulting, mainly as a developer and then project lead for a data warehouse project in the financial sector.
I remember being fascinated by the ambitious company Oda (then Kolonial.no) that wanted to disrupt the Norwegian grocery industry. As a student walking around in Trondheim listening to podcast episodes about this exciting development, I did not imagine that I would end up here 5 years later. Luckily, with a solid foundation in SQL and data modeling from my time in consulting, I got to join Oda’s Data Platform team in March 2021.
How does your day-to-day work look like?
We work in flex and focus periods. In focus periods we concentrate on tasks that we have prioritized beforehand, and that are based on our short-term goals. For every four month period we set new OKRs that reflect what we want to accomplish in the period. The OKRs are in line with the strategic direction of the company, but it is up to us as a team to contextualize them. In flex periods we do “everything else” and tie up loose ends. We have currently started a new focus period, where a lot of my focus will be on our data models dealing with customer data. Personally, I thrive when I get to put on my headphones, write code, develop data models, test them, and iterate until I have a final product — models that behave in the way we want them to. I’ll elaborate on the work we are doing on the customer data models in the “What are you working on right now?” section.
Since a part of our team’s mission is to enable others, we have a rotating support role where the team members take turns being the “Data Platform support person” for one week at a time. This responsibility entails being present in our support channel on Slack, answering questions, and doing ad-hoc requests. We also take extra responsibility for ensuring that our scheduled jobs run successfully so that end users have fresh data. That setup gives us more focus and less interruptions from direct messages in the weeks when we are not the moderator.
What tools and methods do you use?
Our team spans a wide range of areas, and the core competencies of our Data Engineers are for the most part very different. Some of the capabilities we deliver on are:
- cloud infrastructure management and deployment object management in Snowflake, e.g. roles & rights
- custom data pipelines in Argo and support in our self-served ETL tool Fivetran
- standardization of alerting, monitoring, CI, and deployment patterns
- machine learning system standardization (golden patterns in; Dockerfiles, integration with data warehouse, machine learning pipelines, prediction services, CI, ++)
I work a lot in dbt, our data transformation tool, where we define our data models combining SQL with Jinja. For loading data to our Snowflake database, we use Fivetran. Looker is our business intelligence tool. As for data science, almost everything lives in containers on Kubernetes using various resources and services on Google Cloud Platform, and we use open source tools like Argo Workflows and BentoML to help us run pipelines and host machine learning endpoints.
If you’re interested in learning more about the evolution of our insight stack, the decisions made and philosophy behind the tools and technologies chosen, take a look at my teammate Anders’ article; Four building blocks for scaling insights — Part 2: The evolution of our insight infrastructure.
I previously mentioned that most of our team members are Data Engineers. We also have one “potato” that is responsible for Machine Learning Ops, amongst other things. If you’re curious to learn more about ML Ops at Oda, I recommend this podcast episode (in Norwegian) where Kjetil goes in some depth about that topic. If you prefer to read, take a look at the Medium article Empowering End-to-End Data Science at Oda.
What are you working on right now?
When I joined the Data Platform team in March we were in the start phase of building a new data architecture and establishing what we call “the distributed ownership model”. As our company has grown over the years, so has the amount of data and the complexity of our data models. In the distributed ownership model, each team has end-to-end ownership of their own data pipelines and data products (dashboards, machine learning models, etc.). It should be easy for Data Analysts and Data Scientists to extract insight from data by building and maintaining their own data pipelines.
My colleague Lars and I are working on what we call our “core data”. That involves taking responsibility for our most important data like users and orders, and creating a gold standard when it comes to modeling (we use the star schema approach) and guidelines. We want to build data models in a robust way so that both the Data Platform team and other teams can rely on the data and build on top of it, for example for usage in analysis in our BI tool Looker, or in machine learning models. As we iterate we discover new cases that are not yet covered and we adjust our guidelines. One goal of our “core work” is that it will be easier for other teams to make their own models and pipelines, with good performance. We’re still in the early phases and we work closely with other teams, pulling stakeholders into our weekly syncs to get feedback on our work. These efforts contribute to the final goal of creating a robust and scalable Insight Platform fitting the needs of all our product teams.
In order to understand our customers better and build world-class machine learning models we need to work with historical data. This might seem obvious and straightforward, but when you start digging into it, it’s not easy at all to represent the historical state of all our data at any point in time in a correct and easily available format. That’s why our team is creating a framework for modeling and working with historical data. This is of particular interest for Data Scientists, who want to have a single source of truth for their machine learning models. We used customer data to pave the way. A large part of my first months was spent creating a historical customer data model, which can hopefully ease the work for Data Analysts and Data Scientists who want to create their own historical models. This is also an example of how the work of a Data Engineer might differ from that of a Data Analyst; while Data Analysts often have to solve their specific problems fast to keep up with business needs, Data Engineers can take the time to dive into more general, fundamental problems and solve them once and for all, so to speak.
How do you collaborate with your team?
We have a 15-minute stand-up every day where we talk about what we’re working on at the moment, get input on challenges, and raise potential blockers. Depending on what we’re working on we set up ad-hoc work meetings where we sync on tasks or pair program.
We keep track of our prioritized tasks, upcoming focus areas and backlog through GitHub’s project planning tool GitHub Issues. Prior to our biweekly team meeting, we all have a responsibility to go through the backlog and loosely plan what we think should be prioritized the next two weeks. After each focus period, we also have a team retrospective where we talk about what we have accomplished and where we have potential to improve.
What was it like to join a scale-up in rapid growth?
When I joined Oda in March 2021, it was both really exciting and kind of overwhelming. All new joiners get an onboarding plan and a buddy. The first couple of weeks consisted of intros, getting to know our tools, and also my new colleagues. I really appreciated the mix of 1:1 technical hands-on intros and onboarding tasks where I could try things out myself. A part of the onboarding tasks was to have coffee chats with people I should get to know. Especially when starting a new job from my home office during the corona lockdown, I think that was a great part of the first weeks.
All of us who started working at Oda in March participated in joint intro presentations. The first week we talked to our CEO, Karl. At that time we were still called Kolonial.no, but Karl revealed that we were about to get a new name. In addition, we were going international — launching in Finland this fall! I was quite surprised by the openness that was demonstrated, and something that I would say is representative of our culture. It’s a cliché, but I genuinely believe that we have great people throughout our organization. In the coffee chats (that often turned into walk & talks, which I prefer), I felt that everyone was eager to get to know me and to help in any way they could.
Due to the pandemic, we were not able to have a physical team gathering until June, three months after I started working at Oda. We met for burgers and beer, hung out in a park, and finished the night off with some fun and games at our offices in Nydalen. Later that summer we also had a social gathering for the entire Data & Insight (D&I) discipline after presenting what we were working on at the regular D&I all-hands meeting. We were also encouraged to initiate activities on our own, so I gathered a group of people who did a morning bootcamp workout followed by breakfast. I still think the highlight of the year is yet to come, as we are having a semi-traditional Norwegian Christmas party for the D&I Platform in December 💃🕺🏽
I was pleasantly surprised by the flexibility we have in planning our work days. While some of us prefer to go to the gym in the morning or sometimes even during lunch, others need to adjust their schedules for family logistics. As we try to do for our customers, I feel that Oda is doing the same for its employees; creating space for life.
How do you think the Data Engineer role in Oda will look in 5 years?
All of the Data Engineers are currently gathered in the Data Platform team, which is part of the Data & Insight Platform. The Data & Insight Platform also includes our Forecasting team and Insight Enablement team. As the Oda organization grows, so does complexity and the number of edge cases. That’s why our (data) engineers will probably be working on more complex and maybe also profound problems, and will need to be even more specialized. We plan on splitting the Data Platform team into four more specific teams, focusing on different parts of our platform services: Data Platform, Insight Platform, Experimentation Platform, and Data Science Platform. I will probably become part of the Insight Platform (e.g., dbt, Looker, data modeling, etc.) while some other team members will work more with setting up the necessary infrastructure as members of the Data Platform team.
Hopefully we’ll be able to build a solid platform that enables most teams to work autonomously without a dedicated Data Engineer in their team. That said, we expect the engineering complexity and challenges in some of the product teams to increase in such a way that having a dedicated Data Engineer to solve these challenges will be beneficial. As I mentioned earlier, at the moment all Data Engineers are gathered in our Data Platform team. We are now in the process of recruiting the first Data Engineer who will become part of the Marketing Enablement team, part of our Growth Platform. You can read more about what we call “the embedded model” in Four building blocks for scaling insights — Part 1: The embedded model.
It’s very exciting for me to join the company at this scale-up stage, when we’ve just become one of Norway’s first unicorns and upon entering new markets. Hopefully, we’re building the foundation for a robust and globally scalable platform now, so that in five years we will have adapted tools and guidelines to relevant needs and challenges while we can still see that the basic structures remain.
As research has shown, a diverse company is a more successful company. In five years I hope that we’re even better at hiring from the entire population. In particular, I hope to see more women technologists both in Oda and in general. Hopefully, the current 12.5 % women in the Data Platform team will be considerably higher in five years. Our team should also consist of people with a broader range of backgrounds, work experience, and nationalities. As we’re scaling up, we’re planning to ramp up our campus recruitment activity, an initiative I’m really looking forward to participating in!
I hope this article has given you more insight into what the work-life of a Data Engineer at Oda looks like. If you’re considering joining our growing organization, take a look at our job postings. I recommend you to also check out our posts about the Data Analyst and Data Scientist roles.
Thanks to Stian Tokheim, Nina Walberg, Kate Kuzmina, and Siri Bruskeland for their help in preparing this article.
