How to Enhance Performance Using Redis and Diverse Data Structures
Real-world example of using Redis hashes
Introduction
Redis is a robust in-memory data structure store, often used as a database, cache, and message broker. While it’s primarily known for its speed and efficiency in handling in-memory data, Redis also offers persistence repository options, making it suitable for use as a persistent storage solution.
Let’s look at a real-world example of Redis being used as persistent storage. To start with some context, we have several microservices hosted as separate pods in Azure Kubernetes Service. These microservices exchange data and process user requests in asynchronous mode, as the query execution time can be long. The user sends a request that is run for execution and periodically pings the server to ask if the result of the request is ready.
For the fast execution of user requests, we used Redis as persistent storage for each user request execution context, and we improved overall system performance and stability by using a special Redis data type.
Problem and Context
For fast data exchange between the microservices that process the user request, Redis was chosen as persistence storage to transfer the request context. This context is a JSON object size >256 kilobytes; it was not possible to transfer it using a message broker due to the 256-kilobyte limit.
Initially, Redis was used as a distributed cache service that simply stored JSON objects as strings, and the object was updated during the flow execution. An update was implemented by replacing one JSON object with an updated one using a cache key. However, during performance testing of the implemented system, we faced the issue of vast amounts of data being transferred through Redis and reaching the limit of Redis’ overall throughput, which caused Redis’ response time to increase.
There are a few ways to solve such an issue:
- Upgrade the Azure Cache for Redis to the next tier, increasing its throughput, along with the price.
- Use the Redis Modules, such as Redis.Json, to support extended API for working with JSON documents in Redis.
- Use another Redis data structure and its APIs to decrease the amount of data transferred while updating Redis. This was also the least expensive option.
Redis Data Types
Redis’s architecture is designed to handle various use cases, from simple caching mechanisms to complex real-time analytics and geospatial queries.
One of the key strengths of Redis is its ability to persist data to disk, ensuring durability while maintaining high-speed access. This makes it suitable for applications requiring both speed and reliability. Additionally, Redis supports replication, allowing data to be copied across multiple servers for high availability and fault tolerance. This feature is crucial for maintaining service continuity in distributed systems.
Redis also offers advanced features like Lua scripting, transactions, and pub/sub messaging, which enable developers to build sophisticated applications with minimal effort.
This variety of features is based on different data types:
How to use Redis hashes to enhance performance
Prerequisites
To start working with Redis hashes, you need a C# library that implements the Redis client and encapsulates work with the Redis API. In the current examples, we are using StackExchange.Redis installed as a Nuget package.
Code examples
To solve the problem described in the example above, we are using more than just simple strings. Each entity we store and update in Redis will become an array of hashes. Refer to the following implementation examples:
First, we will define the interface that should be implemented. This interface must support storing in Redis as a collection of hashes and also reading a collection of hashes to the entity object:
public interface IHashEntryConvertible<out T>
{
T ConvertFromHashEntries(HashEntry[] entries);
ICollection<HashEntry> ConvertToHashEntries();
}NOTE: This is only one example of the options that can be implemented. You could use Reflection, or another system of your choice, but explicit interface implementation for each type is the best solution when you want to have fast write and read operations, and when the count of your entities is not more than 10.
Next we create the an entity, which implements the interface declared above:
public record OrderProcessingItem: IHashEntryConvertible<OrderProcessingItem>
{
public Guid OrderId { get; init; }
public string OrgKey { get; init; }
public string UserKey { get; init; }
public string PostBackUrl { get; init; }
public OrderProcessingItem ConvertFromHashEntries(HashEntry[] entries)
{
var entriesList = entries.ToList();
return new OrderProcessingItem
{
OrderId = Guid.Parse(entriesList
.Find(x => x.Name == nameof(OrderId)).Value),
OrgKey = entriesList.Find(x => x.Name == nameof(OrgKey)).Value,
UserKey = entriesList.Find(x => x.Name == nameof(UserKey )).Value,
PostBackUrl = entriesList
.Find(x => x.Name == nameof(PostBackUrl)).Value,
};
}
public ICollection<HashEntry> ConvertToHashEntries() =>
new List<HashEntry>()
{
new(nameof(OrderId), OrderId.ToString()),
new(nameof(OrgKey), OrgKey),
new(nameof(UserKey ), UserKey),
new(nameof(PostBackUrl), PostBackUrl ?? string.Empty)
};
}The final step is to implement a services that allows for operation creation, retrieval, and updating on the Redis hashes.
public class StorageService<TConfig> : IStorageService
where TConfig : ICacheConfig
{
private readonly IDatabase _database;
private readonly TConfig _cacheConfig;
public StorageService(IOptions<RedisConfiguration> redisConfiguration,
TConfig cacheConfig)
{
_cacheConfig = cacheConfig;
if (redisConfiguration != null)
{
_database = ConnectionMultiplexer
.Connect(redisConfiguration.Value.ConnectionString)
.GetDatabase();
}
}
public async Task<T> GetValueAsync<T>(string key,
CancellationToken cancellationToken = default)
where T : IHashEntryConvertible<T>, new()
{
var hashEntries = await _database.HashGetAllAsync(key)
.ConfigureAwait(false);
return hashEntries?.Length > 0 ? new T()
.ConvertFromHashEntries(hashEntries) : default;
}
public async Task<T> GetFieldValueAsync<T>(string key,
string field, CancellationToken cancellationToken = default)
{
var fieldValue = await _database.HashGetAsync(key, field)
.ConfigureAwait(false);
return fieldValue.IsNullOrEmpty ?
default :
SerializationHelper.Deserialize<T>(fieldValue);
}
public async Task SetValueAsync<T>(string key,
T value, CancellationToken cancellationToken = default)
where T : IHashEntryConvertible<T>, new()
{
await _database.HashSetAsync(key,
value.ConvertToHashEntries()
.ToArray())
.ConfigureAwait(false);
await _database.KeyExpireAsync(key,
TimeSpan.FromSeconds(_cacheConfig.DefaultExpirationTimeInSeconds))
.ConfigureAwait(false);
}
public async Task SetFieldValueAsync<T>(string key,
string field, T value,
CancellationToken cancellationToken = default)
{
await _database.HashSetAsync(key, field,
value.GetType().IsEnum ?
value.ToString() :
SerializationHelper.Serialize(value))
.ConfigureAwait(false);
await _database.KeyExpireAsync(key,
TimeSpan.FromSeconds(_cacheConfig.DefaultExpirationTimeInSeconds))
.ConfigureAwait(false);
}As these examples show, we can set data to Redis as a JSON object, as well as get or set values for object fields. This is possible because we store objects as collections of hashes and can operate each field separately.
Conclusion
Redis, a distributed cache and persistent storage solution, offers significant advantages for backend developers and data engineers. Developers can optimize data transfer and storage efficiency by understanding and utilizing Redis’s diverse data structures, such as hashes. This approach addresses performance bottlenecks and enhances the scalability and responsiveness of microservices architectures.
This article has demonstrated, through practical examples and implementation strategies, how to effectively use Redis to manage large JSON objects and improve system performance. By adopting these techniques, developers can ensure their applications remain robust, efficient, and capable of handling high volumes of data with minimal latency.
Redis’s versatility and powerful features make it an invaluable tool for modern distributed systems. Mastering its advanced usage can significantly elevate a developer’s skill set and application performance.
Level up your tech project! Explore our expertise!
