Microservices and the rise of event-driven architectures: New paradigms, new opportunities
The shift to microservices is a paradigm shift that is ushering in another new paradigm: event-driven architecture. These new paradigms are shaping not only how applications are developed and deployed, but also the ways in which those applications enable organizations to process, analyze, and act on data to support new ways of interacting with customers, partners, and other organizations.
To understand these shifts, it’s important to first understand microservices applications and how they differ from traditional applications. Traditional applications are largely monolithic — the application is written as a single unit in which all components of the application are interconnected and interdependent. Until recently, this was by far the most common approach. However, this imposes restrictions on development teams — each time you need to add new functionality, the whole application needs to be rebuilt, retested, and updated. Monolithic applications impose restrictions on operations — they are typically difficult to scale, hard to update, and painful to troubleshoot.
These restrictions have led developers and enterprises to move to microservices-based approaches. The fundamental idea behind microservices is to build applications as a set of small, independent components. This approach has been pioneered over the past decade by companies operating at massive scale like Amazon, Netflix, Facebook, eBay, and Twitter and has been brought to the mainstream with the rise of service-oriented architectures and software-as-a-service applications.
Enterprises are adopting the microservices approach because it brings significant advantages over monolithic application development. By breaking an application into smaller decoupled pieces that are connected via APIs, development teams can work on pieces of the application independently. This means that new functionality can be added and delivered to market more quickly. It also means that new developers can be brought onboard and ramped up more rapidly because they do not need to learn the architecture of the entire application to start contributing. Container technologies like Docker and Kubernetes have enabled and accelerated this change by making it easy to build, test, deploy, and manage microservices, making developers and dev ops teams more agile and productive.
The companies that pioneered microservices are also notable for the scale at which they operate. So it’s not surprising that microservices has played a key role in enabling that scale. Microservices has become the primary choice for scale-out applications because its modularity facilitates easy scale up and down of applications. Unlike monolithic applications, which require that the entire application be scaled up and down at once, microservices applications make it possible to scale individual heavily-loaded components of the application without requiring that other components also be scaled at the same time. That not only makes scaling easier to manage, it also makes applications more flexible and adaptable.
The emergence of event-driven architecture
Microservices isn’t just changing how applications are built, it’s also a key enabler of a new paradigm in how data is processed and used. The most common approach to processing data is one that is a lot like the way that older traffic signals work — no matter what the circumstances at the intersection, those traffic lights wait exactly the same amount of time and follow exactly the same fixed sequence before changing. Batch-oriented approaches to processing data are similarly oblivious to how important, how big, or how fast data is.
Today we’re quickly frustrated when sitting at an empty intersection waiting for a light to change. That same frustration is shared by everyone who’s had to wait for new data to be processed in a batch window that could be hours or even days in the future.
Just as traffic signal systems have become more efficient and productive by using sensors to decide when to change the light, event-driven architecture is an approach to designing applications that makes them more efficient and productive in handling data. Event-driven architecture means designing applications to process and react to data as it arrives, without waiting until the next batch processing window. It’s an approach that industry analysts such as Gartner are talking about and encouraging.
However, event-driven architecture isn’t simply a magical sauce you can pour onto existing applications. Trying to turn a batch-oriented approach to data into event-driven architecture by just processing batches faster is an approach that quickly runs headlong into complexity, performance, and scaling challenges. Event-driven architecture requires taking a new approach, one that supports fast, flexible, adaptive applications.
Microservices are a key enabler of the new approach demanded by event-driven architectures. What’s the connection between event-driven architecture and microservices?
Microservices are by design small components connected by data and actions, making them an ideal fit with event-driven architecture — a response in one microservice is triggered by an action or event in another microservice without the need for human interaction. One microservice publishes an event and one or more other microservices subscribe to that event. The services that subscribe to that event are then able to update their data and their state. This may in turn trigger a new event, which another service or services will subscribe to, and so on.
Microservices also streamline development for event-driven architectures. Individual microservices can quickly be developed, modified, tested, and deployed to keep up with evolving data and uses of that data. Container solutions such as Docker and container management solutions such as Kubernetes commonly used to support microservices provide deployment management, resource management, and scaling capabilities that allow applications to adapt quickly to changing workloads.
Creating new opportunities
Event-driven architecture is making possible an abundance of new applications across many domains. The internet of things, for example, is still in its infancy, and AI-based applications are just starting to be prototyped. Applications in areas like these are ideally suited to an event-driven architecture.
One key driver of the demand for applications to become real-time and event-driven is the need to develop ever better and more engaging customer experiences. For example, many of the most engaging mobile applications, such as Twitter, Snapchat, Tinder and Shazam, use event-driven architecture to create responsive end-user experiences. A general characteristic of these applications is that a user is often waiting for a response or reactive update, so events must be processed as fast as possible to ensure an engaging end-user experience:
For example, a common flow for the Twitter application is as follows:
- Event creation: a user generates an event, for example writing a tweet.
- Event processing:
a) The tweet must be successfully logged into an ingestion node.
b) The user’s follower list is accessed
c) Filters are applied to the content of the tweet, for example to mark content as sensitive, which requires applying real-time machine learning.
d) If the content is acceptable, then the tweet is replicated (“tweeted”) to followers, requiring fan-out capabilities.
e) Each user’s followers can then interact with the tweet, such as “like”, “retweet”, or tweet a reply back to the originating user.
- Internal event response: “like” and “retweet” are counted in real-time. Advertisement revenue is calculated in real-time based on tweet impressions.
- End-user event response: user sees real-time “likes,” “retweets” and “replies” to create a unique real-time experience that powers real-time conversations via Twitter.
This pattern of event creation, multi-stage event processing, internal event response, and end-user event response is an example of event-driven architecture. Without real-time processing of event data as it arrives, users’ experience of Twitter would be too slow (on the order of minutes) to support the unique experiences delivered by Twitter today.
The Technology Foundation for Event-Driven Architecture
What does it take to make event-driven architecture a reality? The right technology foundation is one of the most critical components.
Event-driven architecture is fundamentally about processing data immediately as it arrives. That requires real-time data processing. Traditional batch or even microbatch approaches simply weren’t designed for this level of responsiveness, and as a result they are simply unable to deliver the responsiveness to data and events that event-driven architecture requires.
Driven by the need for a paradigm shift from batch-driven to event-driven, there has been a huge amount of innovation in the technology available for collecting, processing, and distributing fast, continuing streams of data. Follow us on the Streamlio blog, where we dive into technologies for processing data in real-time, to see our next post, where we’ll dive into the technologies needed to make event-driven architecture possible.