Inside Agoda’s Data Pipeline Team
Introduction
Agoda relies heavily on data to enhance customer experiences and drive business growth. At the heart of our data-driven success are our dedicated data engineers, who are responsible for a wide range of tasks, from data pipeline development to data warehousing. In this blog, Napong Thammanochit shares what it means to be on the data pipeline team at Agoda.
What Our Data Pipeline Team Does
I joined Agoda 10 years ago as a Data Engineer in the Management Information System team (IT-MIS), managing Hadoop systems. Subsequently, I had the opportunity to contribute to establishing the Data Pipeline team from its inception. Over a span of seven years with the Data Pipeline team, I advanced through the ranks as Senior Data Engineer, Team Lead, and Development Manager. I currently have six direct reports.
We are responsible for managing the primary data pipeline of Agoda’s Data Platform. With all servers at Agoda transmitting data through this pipeline, handling 1.8 trillion messages daily, we are committed to upholding data quality, integrity, and accuracy as fundamental aspects of our identity.
Our Tech Tools and Skills
Regarding the Data Pipeline team, our focus is more on coding and server operations than SQL. Although we have engineers proficient in multiple programming languages, our core responsibilities revolve around managing distributed systems such as Kafka and Hadoop. If you’re considering joining the data pipeline team, prior experience in distributed systems is highly valuable, though it’s not a strict requirement. We offer opportunities to learn on the job and develop these skills.
What We’re Working On
We’ve been involved in many exciting projects; one standout among them is our entire pipeline. This pipeline comprises numerous components, encompassing support, validation, and monitoring elements. One notable project was the development of our in-house Schema Registry, a tool tailored to maintain and enforce schema for data within the pipeline. In addition, our Audit system continuously monitors the entire pipeline, proving useful for investigation when issues arise within the pipeline.
Data Engineers vs. Software Engineers: What Makes Us Different
Data engineering and Software engineering require strong programming skills such as coding, algorithm, data structure, database, etc. But Data engineering focuses on designing, developing, and managing data pipelines, processing, storage, and retrieval systems, ensuring that data is available, reliable, and accessible for downstream analysis and use. Also, Data engineering tasks such as data cleaning, transformation, and integration are more specific to the field, whereas software engineering emphasizes system design, coding, testing, and software lifecycle management.
How We Support the Business Goals
Most of Agoda’s analytics data is transmitted through our pipeline to an Analytics Data storage system. This data is then utilized by Machine Learning, Data Science, Business Intelligence, and the source data owners, enabling analysts to obtain accurate results and directly assess their impact on the business.
Teamwork at the Core: Collaborating with Other Departments
As the Data Pipeline team, we provide a service/platform to assist users in transmitting data smoothly and conveniently through our pipeline. We consult with data storage teams to optimize data storage and retrieval processes, maintaining high efficiency and performance.
We also communicate with infrastructure teams to ensure efficient resource allocation and data pipeline deployment, mitigating potential bottlenecks or interruptions.
Staying Ahead: How We Stay Current with Data Engineering Trends
We welcome new ideas from anyone, regardless of whether they are within or outside the data department. Every week, we gather to watch tech videos together during lunchtime. We hold knowledge-sharing sessions every sprint, where team members share their work or experiences.
Advice for Aspiring Data Engineers at Agoda
If you have a passion for working with data, this is the ideal place for you. We handle massive volumes of data that will surely challenge you. Instead of relying on cloud systems, you’ll have the opportunity to design, manage, and resolve issues directly and effectively with real systems, not just from UI. This experience will enhance your expertise in handling data.
Napong Thammanochit, Development Manager, Data Pipeline Team at Agoda
