Universal Dashboarding Platform (UDP) — Data Visualisation Platform at Myntra

Data Products team at Myntra started building a new product once a basic capability to access data was provided to everyone at Myntra in the form of DDP. Next pitstop was creating a data visualisation platform for Myntra which is knows as Universal Dashboarding Platform (UDP) today.

Data visualisation is very important as picture is worth a thousand words

Everyone doesn’t know how to write SQL queries and hence in order to increase the use of data or to democratise data, it is important to provide users with an interface that allows them to interact with data without knowing any technical details to extract data. Users should have capability to slice and dice the data with rich set of data visualisations and create dashboards that can be used regularly.

UDP — Sample Dashboard

Why should we create our own data visualisation platform when so many commercial tools are available in market already?

• Build a generic platform that can serves as data catalog, data api’s and cool visualisation platform for analytics
• Ability to customise the platform as per the needs of Myntra business
• Standardise the basic business metrics and its definitions with a single platform
• Flexibility to scale the platform as business and data grows over time

UDP Overview

UDP comprises of Dashboards and Widgets. Each dashboard is collection of multiple widgets. Widget is independent entity that represents some form of data visualisation. Widgets are resizable and can be placed anywhere in dashboard by drag and drop. Navigation component on the left hand side allows easy accessibility across various dashboards and a filter component on the right hand side allows to filter dashboard data by specific values.

There are lots of features built in:

• Share dashboards with team members, mark your favourite dashboards
• Schedule the dashboards as email reports daily/weekly/monthly
• Export chart widgets as image or export the data in CSV format, ability to even obtain raw data for widget in CSV format
• Compare data for two date ranges, allows to visualise the comparative performance for two time periods
• Explore data allows to play with data and visualisations, very helpful for exploring the datasets without need to create widgets or dashboards
• Business Alerting allows ability to define rules on metrics that will trigger alert, very useful to keep check of drastic drop/increase in any metric
• Data extracts provides capability to get new dataset in the metric store by writing a SQL on top of granular data in the data warehouse
• Custom drill-downs, conditional formatting, derived metrics etc. are the customised features that helps more with data visualisations

Technical Details

UDP is broken down into multiple independent components that helps to achieve more than just having a data visualisation platform.

UDP Architecture Diagram

Backend Services

There are two types of services:

1. Data Services — A generic backend services that serves data based on input parameters and is not dependent on any visualisations. This layer is exposed as REST API which are generically consumed by other applications at Myntra some of which are customer facing applications as well.
2. UDP Services — A service layer that saves users interaction with visualisation interface. This layer is also acting as a bridge between UI and Data Services and performing very thin semantic changes on the data passing through this layer.

• Java, Tomcat, Jersey, Spring & Hibernate for implementing REST web-services and application service layer

Data Caching

This acts as one of the key component to help provide faster data by minimising the number of database queries. Cache hits are generally served in milliseconds and database hits take few seconds depending on the type of data being requested.

• Redis is used to as data cache. Data gets stored in cache once it is requested by any user and if same data is requested by other user then it is served out of cache itself. Data is cached by generating a unique key based on request parameters of data services API

A pre-cacher is implemented that scans through all the dashboards created by user or accessed by users in last X days and it caches the data with default date for all the dashboards. This ensures super fast first load of any dashboard being accessed by user.

User Interface

• Unity Framework of Myntra that uses ReactJS and NodeJS for user interface

Metric Store

Data in the metric store is structured in form of collections, metrics and dimensions. Each collection stores the data of co-related metrics at specific dimensions. User can also define derived metrics by defining operations on metrics.

UDP can support any ANSI SQL compliant database as metric store

• AWS Redshift as Metric Store

Metadata (Data Catalog)

Metadata store acts as an understanding layer between metric data store and UDP. The metadata store is responsible to define collections, metrics and dimensions mapped to a table in SQL world. It also supports definition of each metric that is eventually displayed to end user of UDP. It supports various additional features like formula definitions, type of metrics (e.g. Aggregate Metrics Vs Snapshot Metrics), visibility of metric or dimension etc. There are set of REST API defined that enables easy discovery of collections, metrics and dimensions

• MySQL is used as metadata store as well as for application data store for UDP services

UDP Visualisations