Redis Architecture: Unraveling Structures and Design Patterns

The Redis structure contains sentinels, masters, and slaves. 3 main architecture for this.

Standalone

In this architecture, redis runs as a single instance and can be used to store and retrieve data from memory. The single instance is responsible from all the read and write operations of the clients. Suitable for smaller-scale apps that do not require high levels of scalability or fault tolerance.

Master-Slave

A single Redis master instance exist and it is responsible for handling read/write requests. One or more Redis slave instances replicate data from the master and are read-only available. This provides increased scalability, and fault tolerance (because read traffic can be distributed across multiple slave instances), and if the master fails failover can be performed automatically.

Cluster

Multiple Redis nodes are grouped to form a distributed system. The data is automatically divided into clusters, and each node is responsible for a subset of the data. This means the architecture includes several sentinel nodes, masters, and slaves. The cluster architecture’s difference from others is sentinel nodes are responsible to manage the assignments of the instances (master, slaves). Sentinel nodes also provide Sentinel state to monitor the state of all the instances in the cluster that stores information about Redis servers with IP addresses, port numbers, server states. A cluster can have sever sentinel instances. However, each sentinel has the same information as the other sentinel because they share the information with each other. If any sentinel fails another one is available so this provides high availability.

This architecture provides a high level of scalability and fault tolerance because data can be distributed across many nodes. If a node fails, failover can be performed automatically.

Patterns

Bulk Loading

Bulk loading is the process of loading Redis with a large number of pre-existing data. The expectation is to perform this operation quickly and efficiently. Different methods will be used to load data to Redis:

• Pipelining: This is used to send multiple commands at once instead of serially, allowing for sending multiple commands and then receiving a single response.
• Batch Operations: Redis provides tools and libraries for bulk data loading operations, enabling data to be loaded in batches.
• Redis Command Files: Data can be processed in bulk using Redis command files.

Distributed Locks

When developing software, it typically involves a multi-threaded structure. Different operations can be performed simultaneously using different threads. Some operations within the software may work on shared data, and in such cases, there can be inconsistencies in the sources accessing the data. This means that each operation might access the data source at different times, or multiple threads might attempt to work on the same data simultaneously. In such a situation, performing shared write operations on the data can lead to issues.

Assuming there are multiple threads operating simultaneously, a thread will continue its operation based on the initial value it read, without knowing the value that another thread might be using or modifying. This can lead to incorrect operations, known as a race condition. A distributed lock is a method used to control access to shared data by different operations running concurrently on a software system.

For example, consider an e-commerce platform. In this project, product update and product deletion operations might occur simultaneously. If both operations access the same product, and the first user is updating the product while the second user deletes it, data inconsistency will occur. To prevent this, a distributed lock can be used to ensure that only one operation is performed on the data at a time.

Secondary Indexing

Redis stores data in a key-value format, allowing data to be accessed via keys. However, this structure only allows searching based on key values to find a specific value. Secondary Indexing overcomes these limitations by allowing data to be indexed based on other attributes besides the key-value pair. This enables querying based on other attributes rather than just a specific key.

• Hash Structure: The hash structure is used to store object-like data. Each hash can contain multiple fields, and these fields can be used as secondary indexes.
• Set & Sorted Sets: Sets contain a list of unique elements. A sorted set, on the other hand, ensures that elements are kept in order. These values can be maintained as secondary indexes.
• Custom Indexes: Custom indexes can be created within the application. These can categorize hashes or sets using a specific property, enabling faster searches.
• 3rd Party Libraries: Secondary indexing operations can be done more effectively with third-party libraries.

Rate Limiting

An application have public endpoints and want to limit access to the endpoint based on client information (10 request per minutes by IP address). By using the key for each client, the endpoint fetches the value from the cache. If the value is greater than 10, the process breaks. To clear the cache from unused data, expiration is set 1 minute later after the first request.

Pub/Sub

Redis also has the publish/subscribe feature. This allows a client of a slave to subscribe to a channel and receive all messages published to the master. A publisher publishes messages to the Redis and subscribers subscribe/unsubscribe a topic to get messages. Messages are popped instantly to consumers (push-based model). It does not store data, messages are sent out and removed immediately. This pub/sub model cannot deal with large dataset because it’s in the memory data store. Redis does not guarantee ordered delivery, it simply pushes messages to consumers. If a consumer restarts for any reason, there is no way to know whether a message is missing.