RESP-ful Replacements: Redis Alternatives with RESP Compatibility
Redis (Remote Dictionary Server) has been a popular choice for developers due to its speed, high performance, and capabilities as an in-memory NoSQL database and cache. However, despite its strengths, certain shortcomings and limitations might prompt developers to seek alternatives. For example:
- Redis generally runs on a single thread (with a few nuances to consider), so it might not fully utilize multi-core CPUs. This can be a bottleneck for write-heavy applications. Although Redis can handle a high number of operations per second, certain workloads might benefit from multi-threading.
- While in most cases, Redis is not capable of taking advantage of the vertical scaling of its CPU, Redis Cluster can potentially alleviate some of the above concerns by distributing data across multiple nodes, allowing for parallel processing. Clustering can aid in horizontal scaling, but managing and maintaining large clusters can be complex and challenging.
- Redis can face memory shortages during the snapshotting phase, especially when backing up data under a heavy write load. If insufficient memory is allocated, it can lead to a system crash, potentially causing data loss. This problem can also arise during scaling or upgrading of Redis under heavy write load.
- Redis does not natively support multitenancy out of the box in the traditional sense of providing isolated environments for different tenants.
- Redis has transitioned from a permissive license to a dual license, specifically, RSALv2 and SSPLv1 which imposes more restrictions.
It’s important to note that the people behind Redis may not view these as significant, as they have different priorities and a well-defined roadmap. Redis has established itself as a leading key-value cache store, with its creators making decisions based on a deep understanding of the technology and their users’ needs. While alternative solutions might address certain concerns, the priorities of the Redis team may vary.
Although no alternative has addressed all the mentioned concerns, some have attracted attention either by garnering support from significant entities or by resolving certain issues. In this discussion, we will touch upon a few such alternatives, namely Dragonfly, Valkey, KeyDB, Garnet, and Kvrocks which are all compatible with the Redis protocol.
Given that Redis has maintained its position as the industry standard, competitors understand the potential cost of migration. Therefore, many competitors support the Redis protocol, allowing for a smoother transition. The Redis Serialization Protocol (RESP) serves as the communication standard employed by Redis, facilitating seamless interaction between clients and Redis servers. Although initially tailored for Redis, RESP’s versatility extends to other client-server software endeavors. Crafted for efficiency and simplicity, it adopts a binary-safe serialization format, accommodating diverse data types such as strings, integers, arrays, and nested structures. Supporting both individual commands and pipelining, RESP empowers clients to consolidate multiple requests into a single transmission, thereby amplifying throughput and mitigating latency concerns.
Dragonfly
Dragonfly, a newcomer in the cache-store realm, was founded by two ex-Google engineers. Despite its relative youth, Dragonfly has quickly gained traction, amassing an impressive 24K stars on GitHub in just about two years since its launch in 2022.
One of Dragonfly’s standout features is its multi-threaded architecture, a departure from Redis’s single-threaded setup. By leveraging multiple threads to handle incoming requests simultaneously, Dragonfly achieves notably faster performance, particularly in scenarios with high data access rates. Benchmarks suggest Dragonfly can deliver up to 25 times the throughput of Redis on a single instance.
Another significant difference lies in data persistence. While Redis relies on methods like RDB snapshots and AOF (append-only file) for persistence, these approaches can cause latency and memory spikes during data synchronization. In contrast, Dragonfly’s forkless persistence mechanism avoids the memory overhead associated with Redis’s snapshotting process.
Dragonfly offers a Kubernetes native solution. The Dragonfly Operator for Kubernetes provides easy deployment with automated replication and failover for high availability. It also supports over 200 Redis commands, highlighting its compatibility and versatility.
Valkey
A storm hit the in-memory data store scene in 2024 when Redis Labs, the company behind Redis, made a controversial licensing change. Previously, Redis enjoyed a permissive BSD 3-Clause license, allowing for free use and modification. However, the company switched to a dual licensing model. From version 7.4 onwards, it’s under a dual license (RSALv2 and SSPLv1). While still technically open-source, these changes restrict how modified versions of the software can be offered as a service. This sparked a fierce debate within the developer community. Many saw it as a move to restrict Redis and benefit Redis Labs commercially.
In response to this shift, a new open-source project called Valkey emerged. Backed by tech giants like AWS, Google Cloud, Oracle, and others, Valkey is a direct fork of the pre-license change Redis codebase. The Linux Foundation has even stepped in to host Valkey, lending further credibility to the project. This means Valkey currently offers a functionally similar experience to Redis but under the familiar and permissive BSD 3-Clause license. The future remains to be seen. While Redis maintains a larger user base currently, Valkey’s strong industry backing and commitment to open-source principles could see it gain significant traction.
On a side note, let’s appreciate the clever naming of Valkey. It evokes both value-key, which reflects its function in data storage, and Valkyrie, a powerful figure from Norse mythology.
KeyDB
The shift in Redis’ licensing terms left many developers seeking a fully open-source option for their projects that has already demonstrated its robustness and stability. KeyDB steps in as one of the top choices, being a well-known fork of Redis, licensed under BSD-3-Clause. Since May 2022, Snap Inc. has maintained KeyDB, utilizing it for its caching requirements.
KeyDB also benefits from its multi-threaded setup. This design helps KeyDB handle heavy workloads better, especially in setups with multiple threads, as shown in benchmarks where KeyDB outperforms Redis by a good margin.
Thanks to its multi-threaded architecture, KeyDB scales well on systems with multiple cores. It also supports cross-region multi-master replication, making it great for setting up distributed systems. While a single KeyDB node can handle scaling up vertically, you can also scale out horizontally through active-replication or cluster-mode (sharded dataset) for even bigger workloads.
KeyDB also brings some features not seen in Redis. It has Multi-Version Concurrency Control (MVCC) which makes forkless background saving possible, better control over expiration with subkey-level management, and built-in TLS encryption with minimal impact on performance.
Garnet
As Redis began implementing more restrictive licensing, Microsoft Research unveiled Garnet, an open-source (MIT license) cache store engineered to significantly enhance the performance of Modern applications and services. Garnet seamlessly integrates with existing Redis clients, providing extensive support for Redis Commands. A benchmark conducted by Microsoft Research demonstrates Garnet’s superior performance over Redis, Dragonfly, and KeyDB across various scenarios, including high and low payloads and client numbers. Garnet exhibits higher throughput and lower latency in most cases.
Despite being in its early stages, Garnet garners support from Microsoft and showcases promising results, positioning it as a compelling choice for monitoring its development and potentially adopting it in future projects.
Kvrocks
In June 2023, the Apache Software Foundation (ASF) promoted Kvrocks from an incubator project to a top-level project. This achievement signifies that the Kvrocks community has successfully met the ASF’s standards for open communication, diversity, and adherence to “The Apache Way” — a set of best practices for collaborative software development.
Apache Kvrocks is a distributed key-value NoSQL database that uses RocksDB as its storage engine. Kvrocks supports namespaces, which allow you to isolate data for different applications or users. This can help to improve security and manageability. It also supports high availability, which means that your data is stored on multiple nodes in the cluster helping to improve data availability and fault tolerance. Kvrocks is a clustered database. This means that it can be scaled horizontally by adding more nodes to the cluster, to improve performance and capacity.
And More…
Many initiatives are underway to enhance the support for the Redis protocol and augment the functionality and efficiency of Redis. Even databases that do not inherently support Redis functionality are offering gateways, facilitating seamless interaction between Redis applications and these databases. For instance, Skyhook serves as a proxy server situated between the Redis client and the Aerospike database. It interprets Redis commands into Aerospike operations, ensuring minimal impact on application code. Additionally, TidiS, a distributed key-value store built on the robust TiKV engine, offers similar functionality for transitioning from Redis to TiKV. Acting as a compatibility layer, TidiS translates Redis commands into equivalent TiKV operations.
In this article, we’ve introduced several databases and presented some of their reported benchmarks. I strongly recommend reviewing the sources to understand the specific conditions under which these benchmarks were conducted. Nonetheless, it’s crucial to approach these benchmarks with a degree of skepticism, as they often depict situations that advantage the respective product. For example, in this post, Redis has contested the benchmarks that Dragonfly has presented believing “the comparison does not represent how Redis is run in the real world”. Conversely, in this thread, it appears that some individuals hold a contrary view, suggesting that Redis is over-engineering and making excessive modifications to achieve improved outcomes in that benchmark. Clearly, it’s important to exercise caution when accepting claims from any party.
While Redis continues to enjoy popularity as a go-to solution for in-memory data storage and caching, an expanding array of alternatives has emerged, promising heightened performance, scalability, and functionality while remaining compatible with the Redis protocol. Whether the objective is to achieve multi-threaded performance enhancements, attain distributed scalability, minimize latency, or access specialized features, the landscape offers a wealth of options tailored to specific use cases and requirements.
Exploring Redis protocol-compatible alternatives presents an opportunity to unearth new avenues and refine infrastructure for heightened efficiency and performance.
Choosing judiciously, may your journey in cache store optimization be marked by swiftness, scalability, and seamless integration.
