Event Driven Architecture to Deliver Business Innovations
Deliver Faster & Better Integrated Financial Services to Customer & Business Partners
Introduction
Indonesia Digital Economy will push economic growth to estimated $150 Billion by 2025 has driven many new business demands and new IT innovations to fulfill that demand. (source: McKinsey).
We believe that new business demands that have become driving factors in Indonesia Digital Economy are:
- Faster Products/Services delivery,
From food, ride sharing, travel, e-commerce order to financial transaction such as interbank transfer, payments; must be delivered almost instantly, even few minutes delay is sufficient to drive customer away. - Cheaper and Better Products/Services,
Customer demands on cheaper better products/services has pushed business to rethink to deliver new products with lower COGS that can help them lower sales price to offer more competitive products/services while at the same time maintaining a healthy Gross Profits. - Convenient & Secure Payments,
Wide range of payment instruments from QR Payments, Stored Value (Bank Indonesia term: Uang Elektronik), Digital Wallets, Direct Debit, Credit Card, and Direct Transfer to CASA/Virtual Account can fulfill wide range of payment use cases that has become financial backbone for modern Digital Economy. - Smarter business that understands customer better,
Customer expects businesses to understand their preference better, by offering the right product with the right price at the right time. Business that can deliver this will reap benefits in the form of loyal customers, high repeat orders, and high customer satisfaction that will bring new customer into their business.
This article will focus on the first business demand: Faster Products/Services delivery, and how IT innovation can fulfill that demand.
Challenges
Faster Products/Services Delivery business demand has multiple IT complexities; they are:
- Near Real Time Integration with Business Partners/Suppliers
Use Case: Payment Notification to Business Partners/Suppliers Backend System that payment for particular order/invoice has been credited to bank account successfully, which will trigger Products/Services delivery to customer. - Near Real Time Integration with Customer Channels
Use Case: Notification to Mobile Device via SMS or Mobile Push Notification or Email that certain financial transaction has been completed successfully
We use Near Real Time terminology where integration with multiple system can be delivered in less than 1 second, which is fast enough to fulfill business requirements.
To implement Near Real Time Integration, we evaluate several options:
- Strongly coupled integration,
Our Banking Applications will know exactly surrounding systems (internal or external systems) that will be integrated. This approach is usable when we know exactly how many and what type of systems that will be involved in the integration scenario.
However, in complex system where multiple new system being integrated every year, this option will be very hard to implement, since we can’t predict new systems in the future that will be integrated with our Banking Applications. - Loosely coupled integration,
Our Banking Applications do not know exactly surrounding systems (internal or external systems) that will be integrated. This approach provides benefit where our Banking Applications do not need to change when new systems being integrated in the future.
Based on options above, we choose to implement Loosely coupled integration strategy where we believe it will survive future complex business integration, and it will provide us the agility to support business innovation that can deliver new business and drive more revenues or reduce operation cost due to efficient business integration.
To implement Loosely coupled integration strategy, we reviewed multiple Integration Architecture patterns, and we fall in love with Event Based Architecture patterns due to several reasons. Before we start explaining further, we will explain terminologies that we will use in this article.
Event Driven Architecture Building Blocks
Based on this problem we look at Integration Patterns reference and identified that Event Driven Architecture has the potential to solve our problem, highly simplified conceptual architecture to increase clarity shown below. (source: IBM & Wikipedia).
Let’s review each how each component interacts.
Events
Events are notifications of change of state. Notifications are issued, or published and interested parties can subscribe and take action on the events. Typically, the issuer of the notification has no knowledge of what action is taken and receives no corresponding feedback that the notification has been processed.
- Events are notifications of change of state that has business value
- Events are records of something that has happened in the past
Events Implemented as application message in various formats (XML, JSON, Avro, CSV) transported using various options (http, sftp, smtp, raw socket).
Example:
- Fund has been received
- Payment has been completed
- Payroll file has been processed successfully
- Debit / Credit Card Issued / Blocked
Event Producer
System that generates events, after business process completed. This system will continuously generate events delivered to Event Backbone, providing continuous business events that can be broadcasted to any relevant System who need to take action based on specific event.
Example:
- Core Banking System
- Credit Card System
- Treasury System
- Mobile Banking Channel
- Data Warehouse System
At the very minimum business events must have information that can be useful for event consumer to take action, consists of Event Source, Event Name and Event Information.
Event generation process is mechanism to publish event in a form of message to Event Backbone.
Example:
Several approach on how to generate business events:
- Application generated events
This approach is feasible when we have access to application source code.
Implementation example: Event generation code can be placed within business logic using Dependency Injection Framework.
Pros:
Very flexible, we can generate any events in many areas of business logic processing
Cons:
Application must be modified, rebuild, retest and redeployed. - Data generated events
This approach is feasible when we don’t want to change application or don’t have access to source code. Event generation will be placed within database server, either by having a code running inside DBMS instance (trigger based), or by having an application that constantly monitor database changes by reading database transaction logs (Change Tracking & Change Data Capture).
2.1. Trigger based
Event generation code is placed within table triggers (after insert/update/delete).
Pros:
Applicable to any DBMS that supports table triggers.
Cons:
Event generation code is part of database transaction, therefore will increase overall database write processing time, and must be done carefully to ensure it will not introduce another problem such as scalability issues.
2.2. Change Data Capture
Event generation will be published by an application that continuously tracking database transaction logs, and raise an event when specific table rows/columns got inserted/modified.
Pros:
Non-intrusive process will not affect the DBMS processing, since this application is reading database transaction logs that contain past transaction, and will not interfere with in-flight database transaction.
Cons:
Commercial offering is not cheap. As an alternative there are several open source options available with growing support from the community.
Example Change Data Capture Application:
- Open source: Debezium, LinkedIn Databus
- Commercial: Oracle Golden Gate, IBM InfoSphere Data Replication, Microsoft SQL Server CDC
Event Backbone
The event backbone is the communication layer in the event driven architecture. It becomes the Publisher/Subscriber communication platform between Event Producer and Event Consumer.
At this high level we would consider two types of relevant technologies for the event backbone, Message Brokers and Event Logs. Both technology types could be used to achieve the event communication style, with the “Publish and subscribe” model however, it is also important to consider other capabilities which are frequently used within event driven solutions:
- Keeping an Event Log as a time sequenced as it happened recording of events (Source of the truth).
- Enabling direct replay of events.
- Enabling Event Sourcing as a way of recording state changes in distributed systems.
- Enabling programmatic access to the continuous event stream.
Example Event Backbone: Apache Kafka, ActiveMQ, RabbitMQ, IBM MQ, Oracle AQ, TIBCO EMS.
Event Consumer
The event consumer reacts to the event delivered by Event Backbone. An event consumer can be as simple as updating a database or business dashboard, or as complex as required, carrying out new business processing as a result of the event. Event Consumer can be an application within one organization, or outside organization — for example suppliers, or business partners.
Example:
- Partner System
- Payment System
- Mobile Banking Backend
- Delivery Order System
Event Consumer in recent years have been developed using Microservice architecture, although it can also develop using Monolithic architecture.
Primary drivers for adopting Event Consumer in Microservice architecture are:
- Highly maintainable and testable
- Loosely coupled
- Independently deployable
- Organized around business capabilities
- Owned by a small team
For those who choose to implement Event Consumer in Microservice Architecture, may choose container-based platform as their deployment model to achieve advanced capabilities such as self-healing, fault isolation, independent service deployment, highly scalable, and optimum resource consumption.
Implementation
As stated at the beginning of the article, our primary business driver when we choose to implement Event Driven Architecture is to deliver Faster Products/Services Delivery, and we have identified several businesses uses cases, and in this article, we choose on 1 use case below.
Use Case: B2B Payment Integration
Background
Our business partners require to deliver order to their customer in time efficient manner. Currently, they are monitoring dozens of their Account Receivables account for payment transfer transaction, and correlate payment transfer with specific order from specific customer. When their customer order and payment transfer increased, this approach is not scalable and it prevents their business from growing.
Solution
We choose to implement Event Driven Architecture, because the solution will be applied to many business partners, and we don’t want to have tightly coupled integration to each one of them, since we expect as Indonesia Digital Economy grow in the future, there will be many more business partners/companies will have benefit by having real time B2B Payment Integration with Bank Mandiri. Therefore, our integration approach must be loosely coupled which can be delivered using Event Driven Architecture.
Implementation
We will try to walk through how we implement the solution, by evaluating each Event Driven Architecture Building Blocks.
Events
We will publish new Event, let’s call this event as “Payment Received”. This event will be published when payment transfer has been received into our Business Partner Account Receivable bank account, and we provide with sufficient information to allow them to correlate payment with specific order.
“Payment Received” event is implemented as an outbound API with one standardized format.
Event Producer
We choose to use Change Data Capture Technology to generate “Payment Received” event, to ensure no change required at the business application and therefore minimize business impact and to avoid time-consuming regression test.
Event Backbone
To ensure highly scalable Event Backbone, we choose to implement Apache Kafka, since we believe this platform is highly scalable and capable handling thousands of events per seconds, publishing to hundreds of Event Consumer system.
Event Consumer
We implement “Payment Received” as an outbound API, that will be pushed to specific API Endpoint belongs to individual Business Partner.
Our Business Partner will responsible to have an API Endpoint, and implement “Payment Received” outbound API to trigger their own internal payment processing.
Although we expect our Business Partner API Endpoint to be always available, when it is not available, we will keep the undelivered outbound API in persistent storage. When API Endpoint becomes available, we will send undelivered outbound API in a FIFO manner to ensure transaction sequence will be delivered in original transaction sequence.
Conceptual Architecture B2B Payment Integration
Highly simplified Conceptual Architecture from previous explanation, where we believe this architecture can support future business innovation where thousands of Business Partners becoming integrated with our system.
Business Benefit
With B2B Payment Integration in place, our Business Partner can receive payment notification in near real-time, allowing them to scale their business and deliver their customer order in a more time efficient manner, also eliminating countless hours of manual labor work on their Account Receivable department.
This innovation is one of many of our Digital Banking Innovation, to support the growth of Indonesia Digital Economy, delivered as a realization one of our key culture: Bersama Membangun Negeri.
What Next
We are revamping many of our applications to support Event Driven Architecture, there are several potential use cases that we evaluating at the moment such as near real-time next best offer, near real-time data warehouse, we will share implementation in other part of our system in the next article.
Thanks for reading!
References:
- https://www.mckinsey.com/~/media/McKinsey/Locations/Asia/Indonesia/Our%20Insights/Unlocking%20Indonesias%20digital%20opportunity/Unlocking_Indonesias_digital_opportunity.ashx
- https://ibm-cloud-architecture.github.io/refarch-eda/concepts/terms-and-definitions/#event-sources
- https://en.wikipedia.org/wiki/Event-driven_architecture
