Introduction
In the digital age where grocery, pharmacy and fashion are all delivered to customers via web and mobile apps, the ability to harness real-time data is a game-changer for businesses. The data engineering team at Loblaw Digital has developed two powerful tools based on this new paradigm, both fed by their Data As A Service framework:
- The Product Domain Aggregator, that leverages cloud technologies like Google BigQuery, BigTable, DataFlow, and PubSub to deliver real-time product metadata to drive important functions such as Search and Recommendations.
- The Order Aggregation Service, which ingests real-time order updates from Google Firestore and pre-calculates a number of useful aggregation metrics to enable advanced customer loyalty and retention programs such as voucher codes for new customers and promotions that unlock new deals to be used later.
In this article, we will discuss these solutions in detail.
Product Domain Aggregator
The Product Domain Aggregator (PDA) is a metadata aggregator that is fed by a number of real-time data sources. These include the Product Catalog Service which provides general product metadata, the Pricing and Promotions Engine which provides pricing metadata, and the Helios Inventory Service which provides inventory metadata. The downstream consumer of this metadata is the Helios Recommendations Engine (HRE), which recommends products to customers based on their current and recently viewed products.
Real-Time Data Ingestion
The data from each source is ingested in real-time into both BigQuery and BigTable. This two-pronged approach, based on the Lambda architecture, is fed by an automated and customizable data ingestion and delivery framework known as Data as a Service (DaaS). The data that lives in BigQuery is used as a historical record, and can be queried by reporting teams and included in derived tables.
Delivering the Data
A subset of the data model required by the downstream consumer is created by extracting from and transforming the source data model. This downstream data model is what is used for delivery to BigTable, where it is queried by HRE through an API. Every time new data is ingested, it overwrites the previous version of that data from that particular source. This ensures that the product metadata stored in BigTable is always an up-to-date snapshot of all product metadata. BigTable was chosen for this task because it is optimized for a large amount of activity and fast read times, allowing HRE to make thousands of calls to the PDA API per hour.
Recommendations on JoeFresh.ca
When browsing an item on Joe Fresh, there are two carousels on the bottom of the page. The “You May Also Like” carousel is generated by HRE and uses machine learning algorithms fed by PDA and DaaS in order to deliver optimized recommendations based on a customer’s previously viewed history and similar product metadata. This allows customers to quickly find more products that might be exactly what they are looking for!
Order Aggregation Service
In addition to the PDA, the data engineering team at Loblaw Digital has also developed the Order Aggregation Service (OAS). This service uses the same DaaS framework as the PDA, but it reads order data from Google Firestore into BigQuery. It then calculates and loads the number of fulfilled orders per customer and additional useful metrics into BigTable. This data is exposed by an API and is read by the Fulfillment as a Service and the Pricing and Promotions Engine teams.
Conclusion
The Product Domain Aggregator and the Orders Aggregation Service are prime examples of how cloud technologies can be leveraged to deliver real-time data. By harnessing this power, businesses can stay ahead of the curve and deliver a superior customer experience.
This is the end of our three part exposé on Data Engineering at Loblaw Digital. We hope to deliver you more fresh ideas soon. Thanks for reading!
