How We Handle Millions of Invalidations-Part 1

At the moment, we have more than 20 millions different sellable products in Trendyol. At any time, any product can be sold, new products can become sellable, new campaigns may be opened, new promotions can be applied. Therefore, product data can be changed at any time. We have to apply these changes to products as soon as possible. Otherwise, we could display wrong information about a product and even we could sell a product with a wrong price. As Trendyol tech team, we call these product data changes as “content invalidation” and we are able to handle millions of invalidations every day and in these article series we are going to describe how we handle these millions of invalidations without affecting user experience.

Figure 1: Invalidation Process

We can describe content as product metadata like name, description etc. and all of its related information like price, stock, promotions etc. As Trendyol technology team, we store content data in two different data sources for different purposes. We use ElasticSearch for our search services and campaign detail service which provides search functionality inside a boutique. We also use Couchbase buckets for our product detail service. We should apply each invalidation to both our ElasticSearch and Couchbase data sources in order to be consistent at each page of Trendyol website and application.

Content Invalidation Service

Content invalidation can be in different ways. For example, the price or the stock of a product may change etc. Every invalidation type has different significance. We have to apply price invalidation as soon as possible whereas we can postpone stock invalidation.

Each invalidation starts with an invalidation message. These messages are sent to different Rabbitmq queues based on their invalidation types. Invalidation messages can be in different formats but all of them have either a content id or a listing id. Content id is the unique identifier of each content. Listing can be basically described as the combination of merchant, size and fulfillment type of a content. Each product can be sold by multiple merchants and each listing contains product size and merchant information. For example, if a merchant with id 100 sells S and M size of a product and another merchant with id 200 sells the same product’s M and L size, we store four listings and each of them contains merchant and size information. The details of the listing is out of context for this article. Listing id is the unique identifier of each listing. We can find the content id of the content which has a listing with the given listing id. We are sure that a listing can be in only one content thus an invalidation message with a listing id will invalidate only one content. How we find the content id from a listing id is again out of context for this article.

Figure 2: Invalidation flow from Rabbitmq queues to Kafka topics

Content Invalidation Service (CIS) listens each Rabbitmq queues to get invalidation messages. Each invalidation message is transformed to a different type of message and sent to a different api called Content Invalidation Api (CIA). The details of each queue is described below.

Management of Assortment Changes

Currently, Trendyol website is available in both Turkey and Germany and we are still working for other countries. However, every product is not available in every country. The availability of a product in a country is determined according to the product’s brand and category. If a brand/category becomes available for a country, all of the products of that brand/category become available for that country. Similarly, if a brand/category becomes unavailable for a country, all of the products of that brand/category become unavailable for that country. As Trendyol tech team, we called this brand and category availability as “brand category assortment”. CIS listens this assortment changes and sends them to CIA with reason as “assortment”.

Management of Price Changes

Merchants may set different prices for specific products and specific time intervals. With this way, they can easily apply their discounts early. Our inventory service holds these scheduled prices and default price. If a content does not have any scheduled price, the default price is applied to the product. CIS listens to these inventory changes and sends them to CIA with reason as “inventory”.

Management of Content Metadata Changes

When there is a change in the metadata of the content like name, description etc. an invalidation message is sent to this queue. CIS listens to these changes and sends them to CIA with reason as “product”.

Management of Content Translation

At the moment, Trendyol has two different languages: Turkish and German. We are still working on English website. All of the contents are initially Turkish and they are translated to German and English. Each translated content is sent to this queue with the language information. CIS listens to this queue and sends them to CIA with reason as “translation” and additionally with language information.

Management of Promotion Changes

Merchants are able to define promotions to their products. Whenever a product is created or updated we should apply it to the defined products. CIS listens to all of the promotion change events and sends them to CIA with reason as “promotion”.

Management of Stock Changes

CIS listen to new stock creation events and stock update events. Stock creation events are sent to CIA with reason as “stock”. However, each stock update event may not have the same importance. For example, a stock change from 100 to 99 is not critical, thus it is not necessary to invalidate. By doing this, we aim to decrease the number of invalidations we have to handle. It may be asked how we decide that a stock change is critical. In order to make this decision, we set a critical stock quantity (At that moment its value is 3). When the new stock of a content after the change is less than or equal to this critical quantity, it is more possible to be out of stock and it is decided as critical. We have to display out of stock information about contents to users thus we have to handle these invalidations. We send each critical stock update event to CIA with reason as “stock”.

Content Invalidation Api

Content Invalidation Api (CIA) basically sends content ids to Kafka topics based on the invalidation type. Before diving into the details let’s discuss the invalidation request parameters of CIA.

1. parameterIds: List<Long>
2. parameterName: String
3. isNestedType: boolean
4. nestedPathName: String
5. invalidationType: InvalidationType
6. indexKeys: List<IndexKey>
7. invalidationTime: Long

CIA sends only content ids along with invalidation reason to Kafka topics. However the invalidation request may not have content ids. It can have for example brand ids, category ids, listing ids etc. In order to get content ids, CIA sends ElasticSearch queries and fetches corresponding content ids.

First four parameters are used to create ElasticSearch queries. parameterIds can be content ids, brand ids, category ids etc. parameterName indicates the type of parameter ids. This should be exactly the same name in the ElasticSearch mapping. For example, content ids are stored in “id” field in ElasticSearch index, therefore parameterName should be “id”. Some of the fields are nested fields in ElasticSearch mapping. isNestedType indicates whether this parameter is nested or not. If it is nested, CIA sends a nested ElasticSearch query. nestedPathName gives the nested path. It is used only if isNestedType field is true. The details of nested types and nested query can be found at this link.

InvalidationType is an enum used to get the reason of the invalidation. This enum also holds topicGeneratorKey which is used to find the corresponding Kafka topic. The possible values are listed below:

• PROMOTION
• ASSORTMENT
• BRAND_CATEGORY
• SUPPLIER
• TAG
• LABEL
• CONTENT_SCORE
• RATING_SCORE
• FIT_RATIO
• STOCK
• PRICE
• INVENTORY
• PRODUCT
• CAMPAIGN
• CONTENT
• IMMEDIATELY
• TRANSLATION

Currently, Trendyol website is available in Turkey and Germany. We are still working for different countries with English language. IndexKey is an enum used to indicate language and country of the invalidation request. Turkish contents and other contents (called international content) are stored in different ElasticSearch indexes. This enum also holds this index information. Currently available values are listed below:

• trTR
• deDE
• enAT
• enBE
• enBG
• enHR
• enCZ

Finally invalidationTime is a unix timestamp of the request time. It is used when an exception occurs. The details will be given later.

When a request is sent to CIA, it first finds the content ids which will be invalidated. To do that, it checks parameterName parameter. If it is “id” then it means that parameterIds parameter stores these content ids. parameterName can be also “brand.id”, “category.id”, “listing.id” etc. In this case, CIA should find the corresponding content ids. To do that, it sends ElasticSearch query by using parameterIds, parameterName, isNestedType and nestedPathName parameters. To which index this query will send is found by indexKey parameter.

After finding content ids, CIA should find the correct Kafka topic to send content ids and invalidation reasons. In order to find the correct topic name, CIA uses invalidationType parameter and indexKey parameter. For example, a request with invalidationType as PRICE and indexKey as trTR is sent to productcontent_price_tr_tr topic, whereas a request with invalidationType as STOCK and indexKey as deDE is sent to productcontent_stock_de_de topic. All of the combinations of invalidationType and indexKey can be sent with a request and each of them are sent to their corresponding Kafka topic.

Exception Handling

It is clear that we should apply each invalidation as soon as possible. However, there can be an exception at any time. We should handle exceptions and still apply invalidations. In order to do that, we follow the steps described below.

1. If there is an exception, CIA takes the request and checks whether the request has invalidation time. If it does not have the invalidation time, CIA set the current time as invalidation time and sends it to Rabbitmq queue.
2. This queue has a ten minutes TTL. After ten minutes, the requests in this queue are sent to another Rabbitmq queue. This queue is listened by our service called invalidation request exception handler.
3. Invalidation request exception handler listens request from this queue and checks whether the listened request is sent in the last 2 hours. If the request is sent in the last 2 hours, it sends it to CIA. With this way, we try to apply these exceptional invalidation requests after 10 minutes from exception time.

Each Rabbitmq queue which is listened by Content Invalidation Service has dead-letter queue. When an exception occurs for a request in CIS, this request is send to corresponding dead-letter queue. We periodically move messages from dead-letter queue to the listened queues. Thus we are still able to handle all invalidations.

Conclusion

In this article, I tried to explain the invalidation process from Rabbitmq queues to Kafka topics. I will continue by explaining how we create content data and how we sync ElasticSearch and Couchbase datasources in Part 2.

Thanks for reading :)