Huddle Diaries — DevOps and Data Platform

By Tania Kohli

This post was originally published on Zomato Blog on May 23, 2019.

Our Huddle on DevOps and Data Platform was one of the events we have been thinking of hosting for a while now. Organisations in this day and age are scaling at an unprecedented pace, and the need of the hour is to develop processes and infrastructure that can effectively cater to quick and quality development.

We won’t deny, this was one of the events where finding the right speakers was harder than usual. That’s also because this specific field is not yet as hot in the Delhi region as it is in the Tech hubs of our nation right now.

It’s also that much harder to find quality people who want to take this up as a career in this region. So our Huddle became a good way to help aspiring engineers to learn more about managing large scale applications and data platforms. We felt it was important to learn from various companies which have faced and overcome their challenges on scalability and data engineering.

The big question we were looking to elaborate upon was — how can we establish an on-going collaboration between effective analysis and efficient infrastructure, that eventually trickles down to better customer satisfaction?

Our learnings from the Huddle

We witnessed representation from some of the fastest growing tech enabled companies Rivigo, Policy Bazaar, 1MG, AWS, GoJek, and many more. The event was also driven by some exceptional individuals and teams, that took us through the journey they have had over time –

• Himanshu Rathore (Senior Software Engineer, Data Infrastructure at Zomato)

Our first speaker Himanshu, took us over the workings of Data Platform at Zomato and how it has evolved over the years. Talking about data as an essential part of Zomato’s DNA, he focused on how it is not only utilised for decision making but also in powering different systems and services.

He spoke about how Zomato’s Data Platform can handle 3.5 terra-bytes of data, processed from over 3 billion events daily. This plays a big part in contributing to the immense enhancements we witness in user experience through Search, Logistics, Machine Learning, Analytics, Operations, and much more.

• Akshat Shah (Systems Engineer, Go-Jek)

Akshat narrated the story of how DevOps has matured in the past 4 years at Go-Jek. In a fast-growing organization, as observed by many, DevOps doesn’t always scale. Hence, a central infra team is formulated, whose core functionality is to create abstractions and toolings for running applications in the cloud, helping developers to solely focus on the business logic.

He went on to elaborate on factors that should serve as hygiene checks for DevOps engineers out there, who wish to effectively manage and execute all tasks at hand — the automation of repetitive tasks, measurement of impact created by each action, and most importantly leveraging the utilization of existing tools to ensure sustainable scalability.

• Suranjan Choudhury (Enterprise Solutions Architect, Amazon Web Services)

Suranjan took us through how EMR can be utilized for processing increasing amounts of customer-generated data in a short span of time, with the least amount of effort and cost expended by organizations.

He focused on the ease of attaining results through AWS tools as compared to setting up systems from scratch, decreasing the initial set up time and ensuring the least amount of manpower is utilized for every task at hand.

• Chinmay Gupta (Site Reliability Engineer, Zomato)

Our final speaker Chinmay explained how Zomato architects the microservice pattern using envoy proxies. He went on to deep dive into the networking layer functions such as retries, circuit breaking, rate limiting, load balancing, etc, which should be taken out of the application layer for maintaining stability and reliability of the system.

Chinmay stressed upon the importance of sustaining reliability within organizations that are developing at a fast pace. Increasing the number of nines in the uptime portfolio contributes to maintaining developer productivity and ensures that efficiency runs through the workings of the entire organization in a big way.

Overall, it was amazing to listen to speakers talk about a variety of topics here. Living in the age of technological advancements, what do organizations really want to achieve with the help of SRE DevOps and Data Platform engineers?

Releases with the least amount of manual intervention and effective collaboration between teams, to ensure swift quality and reliable development.

What was also great about this Huddle was that we saw a lot of aspiring SRE DevOps and Data Platform engineers in the audience, some who were on their way to becoming one and some who were looking to switchover into this space. Some of them just wanted to learn more and apply these learnings to their current jobs, so that they can do more, more effectively. We hope everybody had some key takeaways from the event — which will shape their work, and the positive impact their organisations have on the customers in the long term.

At Zomato, we’re always on the look-out for passionate engineers who take pride in building scalable systems. We invite you to write to us at trialweek@zomato.com, and spend a “trial week” with us at our Bangalore, or Gurgaon engineering offices.

We don’t like the process of an interview — and neither do you. In that spirit, trial week is a no-commitment week that you spend with our engineering team, working on real problems that affect the lives of millions of our customers every day. And if you think (and we think) there is a mutual spark, we would love for you to become a Zoman, and join us in our mission of ‘better food for more people’.