Messaging Systems are Incredible « Teachers » for Software Architecture
If you are a junior dev or software architect, studying the implementation of messaging systems (and other distributed systems) is invaluable.
Why Study Messaging Systems?
Because you will learn A LOT.
A kingdom of tradeoffs
Indeed, these systems offer insights into architecture, trade-offs, challenges, and foundational theories (such as the CAP theorem, split-brain scenarios & quorum algorithms challenges, the impact of efficient serialization formats, fault tolerance & high availability corner cases, network and protocol operations, contention and smart batching, etc).
My advice: study multiple solutions and architecture styles for different use cases.
For example
I will recommend you to study some of these solution below:
- Kafka (Data Streaming Platform): With abundant resources available, Kafka is a great starting point. I recommend Martin Kleppmann’s book, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems.
- RabbitMQ (Queuing messaging system): This system helps you understand a hub-and-spoke (broker-based) messaging solution through a widely used AMQP implementation.
- Tibco RV (multicast legacy): Although an older product, Tibco RV is excellent for learning network basics, focusing on (UDP) multicast, and identifying common flaws and challenges of this “legacy” solution (which has inspired other competitors to do more). Note: I’m not talking about Tibco EMS here which is basically an expensive RabbitMQ-like (AMQP model)
- TREP (formerly RMDS): Explore how to architect a high-available (HA), high-throughput backbone & messaging system for distributing market data in finance, combining numerous patterns and an army of components, cache solutions & protocols (mixing multicast-UDP and unicast TCP)
- Universal Messaging (formerly Nirvana): Study this system to understand the challenges of the “last-mile” distribution using internet protocols, and how to dynamically and smoothly manage access rights, topic aggregations, subscriptions and content publication on the server-side.
- Ultra Messaging (formerly 29West) and Aeron: if TibcoRV was a gen-1 solution for low latency pub-sub messaging system, Ultra Messaging (topic-based deamon-less solution) was the gen-2. You will find lots of nice slides and documentation to learn from. But for a cutting-edge open-source solution, consider Aeron (i.e: the OSS next generation of this kind). It showcases how top engineers like Todd Montgomery and Martin Thompson address issues faced by other solutions, achieving a balance of low latency, high throughput, and reliability like no one else. By the way, any thing coming from these 2 person is worth studying 😉
As a result
As you explore these systems, take time to sketch and to summarize their value propositions, architectures, challenges, corner cases, and pitfalls. This field is a treasure trove of architectural styles, patterns, and trade-offs.
It is even particularly enlightening to understand the limitations of older solutions like Tibco RV and see how subsequent innovations, such as Ultra Messaging, and eventually Aeron, have addressed these challenges, offering more performant, interesting, and modern alternatives.
As a bonus, you’ll likely abandon some of your favored “golden hammers” 🙂 and realize there is definitely no silver bullet.
Ressources
- Topics in high performant messaging (still one of the best tech piece written on these topics)
- Aeron, the open source messaging system for high-throughput, low-latency, fault-tolerant trading systems
- Ultra messaging technical documentation and fundamental concepts
- TIBCO Rendezvous® documentation
- The pragmatic architect (shameless bragging)
- Martin Kleppmann’s book, Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems.
