Design Inventory Availability System using Event Sourcing

In this blog, we will discuss the high-level design of an Inventory and Availability system(I/A) in an eCommerce company using an Event sourcing design pattern. If you are new to event sourcing, please look at this article as a pre-requisite.

Photo credit : Link

Context

An eCommerce company’s critical problem is providing customers with an accurate “Available to Sell” quantity (aka ATS) of products. It is not desirable to capture orders for products which are not in stock OR to display “Out of stock” for the sellable products. This is critical for building a great customer experience on the platform.

I/A system is expected to provide an accurate ATS quantity for billions of products across more than hundred thousand nodes and shopped by millions of customers on daily basis.

Contributing Factors of Inventory Availability

Multiple factors contribute in providing an accurate inventory availability for a product. These factors are briefly discussed below.

1. Node Updates. A Node can be a warehouse, distribution center, or a marketplace center where the products are stocked. Updates related to nodes can impact the inventory availability picture. For Example, if a warehouse is closed (due to Disasters like floods, business decisions etc.), the system should be able to make sellable for all the products as zero. This may apply to millions of products.
2. Catalogue updates. These include updates related to catalogue. For example, whether a product is sellable or not, the associated UPC, GTINs etc. Also the set of eligible nodes where the product is eligible to be stocked. These updates usually come as asynchronous feeds from the Catalogue system. and could be in millions on daily basis.
3. Inventory “Supply” updates. This denotes the “Supply” quantity of a product at a particular node OR the actual physical quantity present in the node to be sold online. These updates usually come as asynchronous feeds from Warehouse, Marketplace systems. There could be hundreds of millions of these requests daily.
4. Customer reservations. Orders placed by customers. There can be millions of product reservations daily on the platform.
5. Fulfilment Updates. Orders being shipped or cancelled from the warehouse also impact the overall inventory availability.
6. Business updates. Business teams may need to hold back some quantity for various reasons. They usually apply the rules at the Product’s category or department level, but this may get translated into millions of updates at each product level finally.

Design

Event sourcing will be a core design principle for creating an I/A system. Like any other system built using event sourcing, this system will have separate read and write portions. Both will have different data models and can be managed and scaled independently.

Writes Path

Any update from the above-mentioned contributing factors will result in a new event being captured in DB. These events will be immutable and will be the source of truth in the system. Events can be partitioned using <product-node> ID in DB.

A command-processor service will process every incoming request (or intent to change the system). This service will perform business validations before writing the new event. For Example, a new reservation can only be captured if ATS > 0.

Event Types

• Reserved event
• InventoryUpdated event
• NodeDisabled/NodeEnabled
• OrderShipped
• OrderCancelled
• ItemDissociatedFromNode
• BusinessRuleApplied etc.

Sample event record in DB

{
    "id": "438483gdee32i329",
    "partitionId": "544366-fn-1234", // Partition key as <product-node> ID
    "eventType": "InventoryUpdated",
    "eventDetails": {
        "productId": "544366",
        "nodeId": "1234",
        "merchantId": "0",
        "onHandQuantity": 33,
        "nodeTypeType": "STORE",
        "gtin": "384031",
    },
    "ts": "2022-12-20T18:38:35.214451+00:00",
    "ttl": 2592000
  ..... other metadata fields
}

Reads Path

Materialised view(s) can be derived from events using the Projection technique. One of these views will summarise information for each <product-node> and get updated asynchronously by applying events. This will contain the actual On-Hand quantity at the Node, total reservations count, business rules, Product enabled or disabled, Node enabled or disabled, etc., and the computed Available to sell (ATS) value. ATS will be a simple mathematical formula comprising the above fields.

For Example, each Reservation event will increment the total reservations count and reduce the ATS.

Other views can also be created using the projection technique which can be used in customer or analytical business requirements.

Sample summary record in DB

{
"id": "fhfryfy4y7t4y75888023u0e",
"partitionId": "544366-1234", //partition key as <product-node> ID
"onHand" : 22,
"activeCustomerReservations" : 6,
"availableToSell" : 16
...other fields
}

Events Streaming

Cosmos DB Changefeed feature can be used to stream events to create materialised views.

Change feed support in Cosmos DB works by listening to an Azure Cosmos DB container for any changes. It then outputs the sorted list of changed documents in the order in which they were modified.

Change feed is an extremely powerful feature and one of the main reasons to be using Cosmos DB as an underlying storage layer in this design. Few DB solutions can stream the DB changes in a reliable and within a single digit latency in milliseconds. In many ways, it resembles Kafka’s consumer/producer and offsets concept. Change feed can also be replayed by resetting the offset/time range and hence critical in Disaster Recovery scenarios! For more details please refer this link.

Summary

In this blog, we discussed a case study of designing a real-world, large-scale distributed system in an eCommerce company using event sourcing design pattern. We discussed how the read-and-write portions of the system could be managed separately to support providing the most accurate inventory availability picture for billions of products on daily basis.