AZ Lamps #20 — Cosmos DB TTL

I keep reviewing the most promising features announced at the Microsoft Build conference. Today’s post is about Cosmos DB Time To Live (TTL).

TTL — is a common approach to managing occupied disk space. You may hear this term when working with caches — from a straightforward single-machine in-memory cache in .NET to more sophisticated distributed caches like Redis or Memcached. When the specified timespan has elapsed, the data is considered expired and can be automatically removed or marked invalid.

The databases world is slightly different because the data is usually long-lived. A relational database with native TTL support is a rare find. And I can understand that — cascade deletes don’t sound like a good idea. Developers prefer to have much more control in implementing something like that.

However, NoSQL databases are the opposite because a document is all-sufficient data. Therefore the majority of NoSQL solutions do implement TTL. AWS Dynamo, MongoDB, Cassandra, etc., have supported the feature for a while, but Cosmos DB was an outsider until now.

It is good to have TTL in place, but I wanted to know how the document removal mechanism correlates with RU consumption. In my previous post, I highlighted that the Cosmos respond with Too Many Requests if you exceed the limit. Bulk DELETEs don’t differ from SELECTs/INSERTs/UPDATEs and consume RU. So is there a chance my app will get stuck because of a cleanup running in the background?

When Cosmos DB runs a TTL-based cleanup and detects a lack of available RUs, the physical deletion of expired items gets delayed. However, expired data will not appear in any queries. That said, we’ve got covered, but I’d still recommend using the TTL feature with Burst Capacity to decrease the chance of RU overflow.

Try this: ask your client about data retention. For how long this of that piece of data should live? Use Cosmos DB TTL (or similar) to apply a policy.