Analytics at lightspeed with Google BigQuery

A fortune 500 company, Aditya Birla Group is one of the largest Indian multinational conglomerates. Today, Aditya Birla Group (ABG) has a far-reaching portfolio — with a presence in the industries of metals and mining, cement, textiles, retail, automobiles, financial services, and telecommunications.

An early adopter of tech, ABG introduced a digital platform to manage the travel of their employees — a fairly new concept in India.

Challenges

Aditya Birla’s Corporate Administration team, headed by Mr. Subrato Sarkaar (Joint President — Corporate Administration) handles and manages employee services support, travel management, services contracting, logistics, transport, real estate, and facility management services. Mr. Sarkaar wanted to revamp and optimize the process of travel management for their employees. It became difficult to streamline and consolidate travel related data from various sources. CloudCover along with ABG created an elegant system to organize and take best advantage of this data by automating everything.

Goals

• At the outset, ABG’s aim was to streamline the data sources since multiple verticals were maintaining data in their respective portals or flat files due to which it was impossible to perform any analytics.
• To create an efficient system that automates all the processes that were earlier performed manually.

Why Google Cloud

Google’s products help in bridging the gap between data and analytics. This allows businesses to take full advantage of their data and use insights that are valuable to their growth. Google’s BigQuery is a great choice when it comes to analyzing data from various sources in a short duration of time.

CloudCover's strategy and approach

CloudCover performed an extensive study of ABG’s existing system and assessed the key challenges. The process of reconciling data from multiple sources was a humongous task and needed to be streamlined. We identified multiple processes which eliminated the deficiency in the existing platforms. CloudCover automated all the tasks that were earlier done manually and built a system that was easy to use. It helped ABG designate responsibilities for each and every department and most importantly, it allowed them to share data in a timely fashion. Since the group comprised of multiple departments, role-based access made operations smooth.

Solution

With the help of Google’s BigQuery and Cloud Storage, we addressed our key challenges. The data was huge and in different formats. It had to be mapped and cleaned. After that it was transformed into JSON format and stored on Google Cloud Storage which served as an input for BigQuery.

The key idea was to NOT have running servers or gnarly SQL for complex string parsing. We focused on massive scalability and that demanded automation.

• A thin client with feature-packed modules written in Angular and hosted on Google AppEngine, a highly scalable app, which allows administrators to grant necessary access to respective vertical team members who can upload data for respective categories.
• It was backed by a highly-scalable database i.e. Datastore, which supports transactions and strongly consistent keys. Google’s Cloud Datastore is a managed, schema-less and highly scalable NoSql database that is especially helpful for developers who want to worry less about scaling their database as usage grows.

The key features were the dashboards for respective members to monitor the overall progress. Various templates were floated as per the processes which were mapped and used during the ETL. The workflow steps consist of the following:

• Using a federated source, read the data and upload it to GCS.
• While reading the data, use a UDF to map/clean/transform it.
• Finally, write the results to a native BigQuery table.

Software Components used for ABG's Data Analytics Platform

BigQuery

A canonical storage with low-latency read and write operations to store analytical data

App Engine

Used for hosting frontend application and ETL

Google Cloud Datastore

A highly-scalable NoSQL database which supports transactions, strongly consistent keys, key distribution, replication and load balancing of data

Google Cloud Storage

Unified object storage for developers and enterprises

BI tool

BIME, an enterprise-grade BI tool for dashboards

Architecture

Results

• CloudCover optimized the process of travel and accommodation management for ABG. Earlier, the system dependency was on manual tasks; due to which, reports could be generated only once per quarter. The new system, however, was much faster and could generate reports within minutes of receiving data.
• As a result of the efficient and automated system, ABG drastically reduced their annual expenses on travel and accommodation.

About CloudCover

CloudCover delivers the insane potential of the public cloud to start-ups & agile enterprises through a combination of weaponized geekiness, extreme automation, and battle-scarred experience.