Dragonfly: 25x faster than Redis
Dragonfly is a drop-in Redis replacement that is high-performance, low-complexity, and built for scale. It is the most performant in-memory data store on Earth.
Want to reach lightspeed when it comes to operating with your Redis cluster?
Ditch Redis… Here comes Dragonfly!
What exactly is Dragonfly?
Dragonfly is an in-memory data store that is 100% compatible with the Redis & Memcached API.
This means it is compatible with all of the same SDKs and tooling you currently use with your Redis instance.
By efficiently using multi-threading & taking advantage of the most recent research — more on than later — the Dragonfly developers were able to reach a 25x higher throughput, improved cache hit rates, and reduced tail latency.
How can it be so fast?
Dragonfly began as an experiment in designing an in-memory datastore in 2022. The team drew from their experience as memory store users and cloud engineers, prioritizing two key features: atomicity guarantees for all operations and low, sub-millisecond latency for high throughput.
To effectively utilize CPU, memory, and I/O resources on current public cloud servers, a shared-nothing architecture was employed.
This approach partitions the memory store’s keyspace among threads, with each thread managing its own “shard”.
The open-sourced library for thread and I/O management powers this shared-nothing architecture.
For multi-key operations, atomicity guarantees were achieved by leveraging recent academic research.
The paper “VLL: a lock manager redesign for main memory database systems” guided the development of Dragonfly’s transactional framework. By combining shared-nothing architecture and VLL, atomic multi-key operations could be composed without the need for mutexes or spinlocks, setting Dragonfly apart from other solutions.
The team’s second challenge involved creating more efficient data structures for the new store.
The core hashtable structure was based on the paper “Dash: Scalable Hashing on Persistent Memory”.
Although originally focused on persistent memory, its concepts were applicable to the problem at hand. The suggested hashtable design from the paper maintained two important properties: incremental hashing during datastore growth and the ability to traverse the dictionary under changes using a stateless scan operation.
Additionally, Dash offered improved CPU and memory efficiency.
What about the future?
Dragonfly 1.0 is a significant milestone, being the first production-ready update… But it’s just the beginning.
The team’s goal is to empower developers and companies to create fast, scalable applications effortlessly. Challenges remain, and they will use the recently-raised $21M capital to evolve Dragonfly, expanding its use cases and handling larger workloads.
In the next phase, SSD storage will be incorporated to extend the main memory while maintaining low latency and reducing costs.
Additionally, they’ll launch a cloud offering for those who prefer managed infrastructure. Sign up for the waitlist to stay updated.
I truly think Dragonfly will replace Redis in the long term, and will revolutionize in-memory storage solutions.
It is better in every single scenario: consumes less memory, has (much) higher throughput, and is soon-to-be 100% compatible with the existing systems!
I will personally use it for my own projects.
I hope you enjoyed reading this Story, make sure to check my other ones, you may find something interesting!
Have a good day 🥳
PS: If you want to read some of my other content, here is a link to my latest Story about the Torrent protocol:
