Event-Driven Architecture Solutions on AWS

Overview of Event-Driven Architecture and real-time example use cases

Introduction

In today’s fast-paced digital landscape, building scalable and resilient applications is crucial. AWS offers a powerful event-driven architecture that enables developers to design highly decoupled and responsive systems. In this blog, we will explore the concept of event-driven architecture on AWS and provide a real-world example use case to illustrate its benefits.

What is Event-Driven Architecture?

Event-driven architecture (EDA) is a modern architecture pattern built from small, decoupled services that publish, consume, or route events. Event-driven architecture (EDA) focuses on the exchange of events between different components or services of a system.

An event indicates a state change or an update. For example: an item placed in a shopping cart, a file uploaded to a storage system, or an order becoming ready to ship.

Events can either carry the state (such as the item name, price, or quantity in an order) or simply contain identifiers (for example, “order #8942 was shipped”) needed to look up related information.

EDA promotes loose coupling between producer and consumer services and enables asynchronous communication between components. This makes it easier to scale, update, and independently deploy separate components of a system.

Key principles of event-driven architecture

1. Loose Coupling: In EDA, components are loosely coupled, meaning they are independent and unaware of each other’s existence. They communicate through events, allowing for flexibility and modularity in system design. This loose coupling enables easier maintenance, extensibility, and scalability.
2. Asynchronous Communication: Events in EDA are propagated asynchronously, decoupling the sender and receiver. This asynchronous nature allows components to process events independently, leading to improved system responsiveness and better scalability. It also enables components to work in parallel, enhancing overall system performance.
3. Event Sourcing: EDA often leverages event sourcing, which involves capturing and storing all relevant events that occur within a system. Event sourcing provides a historical record of actions, allowing the system to be reconstructed or audited at any point in time. It also enables building complex workflows and analytics based on the sequence of events.

Advantages of event-driven Architecture:

1. Scalability: EDA inherently supports scalability by decoupling components and enabling parallel processing.
2. Flexibility: With EDA, components can be added, modified, or replaced without affecting the overall system.
3. Fault Tolerance: If a component fails or becomes unavailable, other components can continue processing events independently. The system can handle failures gracefully and recover without affecting the overall functionality.
4. Extensibility: EDA enables easy integration with other services or systems by leveraging events. New services or functionalities can be added by subscribing to relevant events, making it easier to extend the system’s capabilities without disrupting existing components.

Overview of AWS event services

AWS offers a suite of services that work together seamlessly to enable event-driven communication and processing in event-driven architectures. Here are some commonly used AWS services for event-driven architecture:

Amazon EventBridge

EventBridge is a serverless service that uses events to connect application components and allows to building of scalable event-driven applications.

Amazon Simple Queue Service (SQS)

SQS is a fully managed message queuing service that enables decoupled communication between components in an event-driven system. It allows components to asynchronously send and receive messages (events) through queues, ensuring reliable and scalable message delivery.

Amazon Simple Notification Service (SNS)

SNS is a publish-subscribe messaging service that facilitates the broadcasting of messages (events) to multiple subscribers (components) simultaneously.

AWS Lambda

AWS Lambda is a serverless computing service that allows you to run code without provisioning or managing servers.

Lambda functions can be triggered by events from various sources, such as SNS, SQS, API Gateway, and more. Lambda functions can process events, perform business logic, and interact with other AWS services, making them a key component in event-driven architectures.

Amazon Step Functions

Amazon Step Functions is a serverless workflow service that enables you to coordinate and orchestrate multiple AWS services and custom actions in a visual workflow.

It provides a way to define and execute complex state machine workflows, allowing you to design and manage event-driven workflows with ease.

Amazon API Gateway

Amazon API Gateway is a fully managed service that makes it easy to create, publish, and manage APIs for applications.

Examples of real-time use cases

Microservices communication

This use case is often seen in retail, media, and entertainment websites that need to scale up to handle unpredictable traffic. A customer visits an e-commerce website and places an order. The order event is sent to an event router, and all the downstream microservices can pick up the order event for processing — for example, submitting the order, authorizing payment, and sending the order details to a shipping provider. Because each microservice can scale and fail independently, there are no single points of failure.

This pattern has helped LEGO scale its e-commerce website to meet peak Black Friday traffic.

LEGO e-commerce website

Application Integration

Events allow us to integrate other applications. We can send events from on-premises applications to the cloud and use them to start building new applications. Integrating SaaS applications enables us to create custom workflows using those events. We can use SaaS applications for services like customer relationship management, payment processing, customer support, or application monitoring and alerting.

Taco Bell built an order middleware solution to ingest incoming delivery partner app orders and send them directly to their in-store point-of-sale applications.

Taco Bell SaaS application

IT automation

When running compute-intensive workloads (such as financial analyses, genomic research, or media transcoding), you can have compute resources respond by scaling up for highly parallel processing and then scaling down after the job is complete.

AWS services already generate events, including Amazon Elastic Compute Cloud (Amazon EC2) instance state-change events, Amazon CloudWatch log events, AWS CloudTrail security events, and many others. We can use these events to automate your infrastructure for validating configurations, reading tags in a log, auditing user behavior, or remediating security incidents.

Société Générale automatically scales resources up and down for credit risk analyses.

Let’s discuss the solution for credit risk analysis. The following diagram shows the general architecture of the solution on AWS.

Societe Generale High-level Architecture

Summary of the architecture

• The system is designed to process large amounts of data in a cost-effective and scalable manner.
• The system uses AWS Step Functions to orchestrate the processing workflow.
• Amazon S3 is used for input and output data storage.
• Amazon ECS is used for batch processing.
• Amazon API Gateway is used for real-time processing.
• The system also uses a private subnet for security and a load balancer for high availability.

Description of application setup

1. The application operates within a private Amazon VPC without direct internet access, ensuring network-level isolation.
2. VPC Endpoints are used to connect to all necessary services, maintaining the solution’s isolation.
3. End users interact with the system through a REST API exposed via a private API Gateway.
4. Internal users upload the required data for calculations to a private Amazon S3 bucket.
5. Users invoke the calculation engine of their choice through the private API. Two types of engines are available: real-time (synchronous) and batch (asynchronous).
6. AWS Lambda functions trigger the compute engines.
7. Real-time processing occurs within seconds and utilizes Amazon ECS to run containers on AWS Fargate.
8. Batch processing, which takes longer to complete, is managed through a job queue. End users submit calculation requests via the API and receive a job ID in response.
9. Users query the application through the API Gateway to check the batch job status.
10. The job execution is handled by AWS Step Functions, which acts as a serverless function orchestrator.
11. AWS Step Functions allows us to define the workflow and integrate other AWS services.
12. Amazon ECS tasks are used as the primary execution engine.
13. Preliminary analysis is performed on the input data, including preprocessing and splitting it into smaller parts for independent and parallel processing.
14. The Map state in AWS Step Functions runs each task in parallel, with a configurable concurrency setting.
15. Input parameters are passed to the application as environment variables, dynamically set by the prepare step.
16. Each task runs in an AWS Fargate task (configurable).
17. After all operations are completed, the results of each calculation are gathered into a final state in a private S3 bucket.

AWS Step Functions State Machine Example

Core benefits of the solution architecture

• Removes complex workload distribution logic: Eliminate the code responsible for distributing workloads, as AWS Step Functions will handle this task entirely.
• Utilizes Amazon S3 for file access and writing: Replace any references to the local file system with Amazon S3, allowing access and writing of calculation results.
• The application setup is entirely serverless, eliminating the need for operating system patches or instance maintenance.
• The compute runs are independent, allowing for parallel execution of different engine versions using container tags and separate stage functions.
• This enables faster testing and minimizes the risk of regressions. Additionally, AWS services provide various metrics, such as execution times, failures, and request counts, allowing for comprehensive monitoring and tracking.

Conclusion

In conclusion, event-driven architecture offers several benefits and plays a crucial role in modern application development. By embracing an event-driven approach, developers can design highly scalable, flexible, and resilient systems that respond to real-time events and trigger actions accordingly.

To experience the advantages of event-driven architecture, I encourage readers to explore AWS services like AWS Lambda, Amazon Simple Notification Service (SNS), Amazon EventBridge, and AWS Step Functions. These services provide a robust foundation for building event-driven applications on the AWS cloud. By leveraging these services, developers can unlock the full potential of event-driven architecture and create highly scalable, responsive, and resilient applications tailored to their specific project needs.