Object Store V2 Expiration Demystified

Object Store v2 is an outstanding feature provided by MuleSoft on CloudHub. It has many use cases ranging including storing data as cache, duplicate checks, and watermarking. We can store any number of messages with a max size limit of 10 MB per entry.

While configuring the Object Store, we encounter something called expiration. When you hear the terms TTL and expiration interval, you may wonder, what are these? What values do we need to set? How these are interrelated?

Time To Live (TTL)

Time to live (TTL) indicates how long the data will be available in the Object Store before it is automatically deleted. By default, TTL is set to 30 days or 2,592,000 seconds. If you access the data in the last 7 days before expiry, TTL resets to 30 days. This is called rolling TTL, and it because of this feature, TTL can be considered unlimited. On the other hand, static TTL is reset by updating the same key.

Rolling TTL is the default for Mule Runtime 4.2.1 and later, whereas static TTL is the default for Mule Runtime versions earlier than 4.2.1.

Suppose we want to configure the Object Store to keep values only for 5 minutes. We can do so by specifying the TTL. However, this is not as simple as it seems. To make it work, we need to specify the expiration interval.

Expiration Interval

Expiration Interval is the frequency by which the expiration thread will run. If this value is not set, then the TTL value is ignored and the default 30 days value is used.

Default Configurations

How these are linked? Many of us might have heard that the expiration interval should be less than TTL. But how do these actually work? To understand this, consider the story of the supermarket.

Story of the Supermarket

Once upon a time, there was a supermarket. In the supermarket, there were many dairy products. There was a supervisor whose job it was to remove any product that resided on the shelf for more than 30 days. Not every dairy product has the same shelf life, some expired before 30 days. Since there was nothing mentioned on the products regarding expiry, customers complained about getting spoiled products. It caused the supermarket a setback from customers.

To overcome this, the company introduced the date of expiry (TTL) concept and updated the product packaging to have different shelf lives (i.e., 2 days, 5 days, 10 days, etc.). The date of expiry was in the coded format not visible to end consumers. The supervisor wasn’t given any instruction as to when he needed to check. He had to file a report at the end of each month. He would only check the products for expiry at the end of each month (i.e., after 30 days).

Note: Only the supervisor can read the date of expiry and not the consumers.

Even though each product has its own date of expiry, it was still available for the customers to pick up unless it was removed by the supervisor at the end of the month. The issue of customers receiving spoiled products persisted, even after applying the TTL.

To overcome this, the supervisor was given instructions to check each product and remove any that had expired every morning before consumers start coming to the supermarket . Since the minimum shelf life of any product was 2 days, it could be easily caught by the supervisor and removed from the shelf. In the end, customers were happy and customer satisfaction allowed the supermarket to become profitable.

What if the frequency of checking was 3 days instead of 1 day, which will be greater, the expiration time or TTL?

In other words, the supervisor checks a product on the 3rd day and the product expired one day before. In that case, the spoiled product, which doesn't have much value, will be lying on the shelf and could be used by the consumer causing a bad experience.

To make TTL work, we need to run an expiration thread with a frequency that is lower than TTL to remove stale objects from the Object Store. TTL will not hold any value unless an expiration interval is specified.

I hope this story clarified and simplified the concepts of TTL and expiration interval.

Here is an example configuration with a 5 minute TTL and a 1 minute expiration interval.

Thank you for reading and please share any thoughts or comments.