James Miller talks about how to become a data engineer and what he gets up to in his role at BGL Tech:
I’m a data engineer in the Data Solutions team at BGL Tech. If you don’t know what a data engineer does, it involves designing, integrating and building data from various sources which can then be reported on or analysed by our teams of BI (business intelligence) analysts and data scientists.
How to become a data engineer
My route into data was not conventional. I was training to be an accountant before I moved over to provide application support, for my accountancy practice’s time and billing system, and then for Wickes’ retail systems. These roles were where I started learning about relational databases and SQL coding, which I use daily in my current role. I then moved into business intelligence and data warehousing at Travis Perkins before I joined BGL three years ago, initially as a business intelligence developer in the life insurance part of the business, home to brands such as Beagle Street.
There are people in our team from a range of backgrounds but what we all share is SQL coding experience, familiarity with SSIS (SQL Server Integration Services), an understanding of relational databases and, most likely, another programming language such as Python, as well as the ability to work effectively as part of a team and collaborate.
An agile approach
Before I get into what a typical day looks like, I’ll explain a little about how the Data Solutions team works.
We use the Agile methodology for development as do the other teams in BGL Tech. Our work is planned in two-week sprints.
Before each sprint there is a planning meeting where the team meets to review the tickets (requests for work) raised by our stakeholders and allocates them to a data engineer to work on in the upcoming sprint. There are a whole range of tasks from a variety of different business areas as well as standard tasks, such as monitoring overnight loads from sources such as Google Analytics, transactional systems or external data feeds from third parties (each engineer does a week of system support on a rota basis). Then there is always some ongoing project work that needs to have people allocated to it, for example a review of our pricing data sets or GDPR compliance.
Typical development requests include bringing new data sets into either the operational data store (ODS) or data warehouse (DW), adding new fields to an existing data set, producing data extracts for internal or external use, changing business logic or rules in our processes/ automating processes as well as system maintenance and database analyst tasks to keep the system running well, indexing, backups and restores etc.
A typical day
A typical day will start with a short meeting where I’ll provide a status update to the rest of the team, about the work I did the previous day, what I have planned for the day ahead and any problems or blockers to completing the work I’m doing.
The rest of the day is spent working on the tasks that have been assigned, this might mean writing code, developing an integration process to move and transform data. Talking directly with stakeholders to better understand their requirements and to test and validate work that has been completed.
Last week I took my turn on the support rota and that normally means I’m online a little earlier than normal to check for any job failures that need resolving. The rest of the day is spent monitoring the system and triaging the support tickets before investigating and resolving them.
There are some reoccurring meetings, that are related to agile ways of working, as well as a weekly team meeting to attend which keeps me well-connected to what the priorities are across the team and the wider business.
It’s a role that is fast-paced and brings me into contact with a lot of other areas of the business.
I enjoy writing code and problem solving, and being able to contribute to producing valuable insight that is used to inform business decisions. There is plenty going on across our team, whether that be using data to combat fraud or supporting our early coronavirus response to our customers. Each day brings something new and varied and BGL is an innovative and exciting company to work for.
With over 10 million customers, BGL is a leading digital distributor of insurance and household financial services via brands including comparethemarket.com, Budget Insurance, Dial Direct and Beagle Street. It employs more than 3,000 people across its five sites in Peterborough, Sunderland, Wakefield, London and Paris.
