ZATA: How we used Kubernetes and Google Cloud to expose our Big Data platform as a set of RESTful web services

Authors: Shailu Mishra, Sudhir Hasbe

In our initial blog post about Zulily big data platform, We briefly talked about ZATA (Zulily data access service).Today we want to deep dive into ZATA and explain our thought process and how we built it.

As a data platform team we had three goals:

1. Rich data generated by our team shouldn’t be limited to analysts. It should be available for systems & applications via simple and consistent API.
2. Have the flexibility to change our backend data storage solutions over time without impacting our clients
3. Zero development time for incremental data driven APIs

ZATA was our solution for achieving our above goals. We abstracted our data platform using a REST-based service layer that our clients could use to fetch data-sets. We were able to swap out storage layers without any change for our client systems.

There are three different attributes you have to figure out before you pick a storage technology:

1. Size of Data: Is it big data or relatively small data? In short, do you need something that will fit in My SQL or do you need to look at solutions like Google Big Query or AWS Aurora?
2. Query Latency: How fast do you need to respond to Queries? Is it milliseconds or are few seconds OK — especially for large data-sets
3. Data Type: Is it relational data or is it key value pairs or is it complex JSON documents or it is a search pattern?

As an enterprise, we need all combinations of these. The following are choices our team has made over time for different attributes:

1. Google Big Query: Great for large data-sets (in terabytes) but latency is in seconds and supports columnar storage
2. AWS Aurora: Great for large data-sets (in 100s of gigabytes) with very low latency for queries
3. PostgresXL: Great for large data-sets (100s of gigs to terabytes) with amazing performance for aggregation queries. This is very difficult to manage and still early in its maturity cycle. We eventually moved our data-sets to AWS Aurora.
4. Google Cloud SQL, MySQL or SQL Server: For Small data-sets (GBs) with real low latency in milliseconds)
5. MongoDB or Google Big Table: Good for large scale data-sets with low latency document lookup.
6. Elastic Search: We use Elastic Search for scenarios related to search both fuzzy and exact match.

Key runtime components for ZATA are

Mapping Layer

This looks at the incoming URLs and maps them to backend systems. For example: Request: http://xxxxx.zulily.com/dataset/product-offering?eventStartDate=[2013-11-15,2013-12-01]&outputFields=eventId,vendorId,productId,grossUnits maps to

1. Google Big Query(based on config db mapping for product-offering )
2. Dataset used is product-offering which is just a view in the Google Big Query system
3. Where eventStartDate=[2013–11–15,2013–12–01] is transformed to where eventstartDate between 2013–11–15 & 2013–12–01
4. Output fields that are requested are eventId,vendorId,productId,grossUnit
5. Query for Google Big Query is:

Select eventId,vendorId,productId,grossUnit from product-offering where eventStartDate=[2013–11–15,2013–12–01]

The mapping layer decides what mappings to use and how to transform the http request to something that back end will understand. This will be very different for MongoDB or Google Big Table.

Execution Layer

Execution layer is responsible for generating queries using the protocol that the storage engine will understand. It also executes the queries against back end and fetches result sets in an efficient manner. Our current implementation supports various protocols such as MongoDB, standard JDBC as well as http request for Google BigQuery, Big Table and elasticsearch.

Transform Layer

This layer is responsible for transforming data coming from any of the back end sources and normalizing it. This allows our clients to be agnostic of storage mechanism in our back end systems. We went JSON as the schema format given how prevalent it is among services and application developers

In previous example from Mapping layer the response will be following.

[{“eventId”: “12345”, “vendorId”: “123”, “productId”: “3456”, “grossUnits”: “10”},{“eventId”: “23456”, “vendorId”: “123”, “productId”: “2343”, “grossUnits”: “234”},{“eventId”: “33445”, “vendorId”: “456”, “productId”: “8990”, “grossUnits”: “23”},{“eventId”: “45566”, “vendorId”: “456”, “productId”: “2343”, “grossUnits”: “88”}]

API auto discovery

Our third goal was to have zero development time for incremental data driven API. We achieved this by creating an auto discovery service. The job of this service is to regularly poll the back end storage service for changes and automatically add service definitions to the config db. For example, in Google Big query or My SQL, once you add a view in schema called “zata” we automatically add the API to ZATA service. This way the data engineer can keep adding services for data set they created without anyone writing new code.

API Schema Definition

Schema service enables users to look for all the APIs supported by zata and also view its schema to understand what requests they can send. Clients can get the list of available datasets;

Dataset Request: http://xxxxx.zulily.com/dataset

[
{ “datasetName”: “product-offering-daily”,….},
{ “datasetName”: “sales-hourly”,…………………},
{ “datasetName”: “product-offering “,………….}
]

Schema Request: Then they can drill down to the schema of a selected dataset; http://xxxxx.zulily.com/dataset/product-offering/schema/

[
{ “fieldName”: “eventId”, “fieldType”: “INTEGER” },
{ “fieldName”: “eventStartDate”, “fieldType”: “DATETIME”},
{ “fieldName”: “eventEndDate”, “fieldType”: “DATETIME” },
{ “fieldName”: “vendorId”, “fieldType”: “INTEGER” },
{ “fieldName”: “productStyle”, “fieldType”: “VARCHAR” },
{ “fieldName”: “grossUnits”, “fieldType”: “INTEGER” },
{ “fieldName”: “netUnits”, “fieldType”: “INTEGER” },
{ “fieldName”: “grossSales”, “fieldType”: “NUMERIC” },
{ “fieldName”: “netSales”, “fieldType”: “NUMERIC” }
]

So far, the client is not aware of the location or has any knowledge of the storage system and this makes the whole data story more agile. It is moved from one location to another, or the schema is altered, it will be fine for all downstream system since the access points and the contracts are managed by Zata.

As we rolled out ZATA over time, we realized the need for storage service isolation. Having a single service support multiple back end storage solutions with different latency requirements didn’t work very well. The slowest back end tends to slow things down for everyone else.

This forced us to rethink about zata deployment strategy. Around the same time, we were experimenting with dockers and using Kubernetes as an orchestration mechanism.

We ended up creating separate docker containers and kubernetes service for each of the back end storage solutions. So we now have a zata-bigquery service which handles all bigquery specific calls. Similary we have a zata-mongo, zata-jdbc and zata-es service. Each of these kubernetes service can be individually scaled based on anticipated load.

In addition to individual kubernetes service, we also created a zata-router service which is essentially nginx hosted in docker. Zata-router service accepts on incoming HTTP requests for zata and based on the nginx config, it routes HTTP traffic to various kubernetes services available in the cluster. The nginx config in zata-router service is dynamically refreshed by polling service to make new APIs discoverable.

ZATA has enabled us to make our data more accessible across the organization while enabling us to move fast and change storage layer as we scaled up.

Originally published at https://zulily-tech.com on October 3, 2017.