A Recap of Our Internal Developer Conference: ixConf 2017

Published in

Indix

6 min readFeb 9, 2017

As a company, we’ve always believed in trying our hand out with new technologies, interesting problems and other such activities. Sometimes, these are even accomplished on weekends based on personal interests. We also encourage it by way of quarterly hackathons. While progress on this front is important for us to stay innovative, it is equally important to move ahead as a group by sharing the knowledge across the board. It also helps newer Indixers to understand how Indix evolved and the people behind its successes.

When we were brainstorming ideas to facilitate knowledge sharing, the ixConf was born. The first edition, which happened on 23 January 2017 featured various talks on the evolution of things until now and future solutions. We saw an array of talks ranging from the detailed architecture of the systems running daily behind our APIs to the customer issues we have been able to solve as a team. In a way, these internal opportunities also help teams evangelize their work within the company to gain interest and traction on that topic.

The day commenced with a welcome address by one of our co-founders Rajesh Muppalla, who enlightened us on the need for such events and how we as engineers can contribute back to the community by sharing our knowledge. This year, ixConf featured nine talks. We restricted it to only fellow Indix-ians for the first edition.

Here’s a summary of the varied talks that we got to witness on the day.

The first talk themed “Site Ingestion — The Evolution” was delivered by Arun and Lakshmi. It helped us understand the entire workflow of adding a site to the Indix Cloud Catalog — crawl strategies, parsing techniques as well as how we monitor and maintain a site in the system. We witnessed the transition from ingesting 30 sites a month to 100 sites a month as of now.

Lakshmi spoke about attribute level parsers and how they have helped us better our data quality. This was followed by Bala R, who shared lessons learnt while building a scheduler for our parser systems. Reduction in memory and disk usage and reduced deployments ensured better lifecycle for the parser.

Bala R on parser systems

Venkatesh, one of our ML engineers, gave insights on challenges involved in building large scale classification models. He also enticed us with some updates on what is currently being done to reduce model building time.

Venkatesh

Bala B and Jananee on analytics

Then there were a couple of extraordinary sessions from the team that takes care of serving our large scale product data to end users via the API. Bala B and Jananee, as the first hand spectators as well as initiators of this change, walked us through the evolution and design decisions undertaken during the transition of the analytics systems. The talk enlightened us on how the systems helped us scale to index and serve one billion+ products through the API with lightning response times. This system was designed keeping in mind that it should be able to handle 10 times the data it is serving now. Anitha and Ramesh then spoke about the anatomy of our current analytics system which we call Sherlock, and how we are indexing prices real-time into it and how the tools we have built help us handle backup of the these large indexes in an automated way.

Anitha and Ramesh

The second half of the day was filled with equally interesting talks. Bharath and Bethleheraja described to us the lessons learned in deploying the new API developer portal and the entire transition process from 3Scale to Apigee. They also shared with us their experiences of the D-day production push and how they coordinated the deployment of the API portal which included almost every single team at Indix. Reema’s talk was an attempt to cover all the lessons learnt over the past year on best practices to maximize performance when running Spark jobs and how to avoid common pitfalls.

Reema

Manoj presented an interactive session on MDA infrastructure (MDA is a system that can orchestrate the processing of large volumes of data using a set of configured algorithms and steps) which runs primarily on Kubernetes. Even though we had Mesos infrastructure for deploying apps, we wanted to try Kubernetes as we are always open to trying our new stuff. As a part of this, we also tried out Terraform for infra provisioning and are absolutely in love with it now. This talk also covered the ease with which we handle this infra (we are updating Kubernetes for even minor versions) — we got to know the why’s, the what’s and the how’s of it.

The last talk of the day was by Thiru and Monika, who represented the Customer Success team and gave us insights on different customer requests that are handled by them. The talk also covered the tools and processes that the CS team uses to handle these queries or requests. As a result of constantly being the face of the company for the customers, they also shared valuable feedback on “missing features”, some “common requests” from customers and other such useful nuggets to help improve our product.

The key learning from ixConf was getting a holistic view of the entire data pipeline; the initial design decisions and intentions with which the systems were built, and the champions behind them. We were also reminded how Indix works as a team to solve real-world hard problems.

The day ended with Satya (Indix Co-Founder and CTO) handing out goodie bags to our speakers. ixConf, though the first of its kind, was indeed an enriching event. Going forward, we look forward to hosting such conferences publicly. Watch this space for more and enjoy the rest of the pictures.