Anypoint Platform: Governance Model for Event-Driven Architecture
Event-Driven Architecture (EDA) has become increasingly popular over the last few years. It provides many benefits when dealing with complex integration scenarios, keeping components and systems decoupled, simplifying the evolution of the solution, and allowing for the design of scalable and highly available applications.
However, the complexity of the Governance Model often means that we cannot take full advantage of this type of architecture and its coexistence with other approaches.
Shortcomings of the Current Approach
When we work on EDA, we must define topics, events/messages, data validation rules, and security mechanisms. This might include:
- Sending an email with a template attached
- Registering a ticket in Jira or another similar ITSM tool
- Creating requests via chat in an IT support channel
- Phone calls
All of these options have a common pattern: the use of colloquial language, free text approach to describe what we want to do as part of a specific project/integration.
Therefore, there was no standard procedure to describe how we want to enable an EDA, much less facilitate asset reusability and the promotion of synergies between projects.
As a result of this lack of governance and standardization, it is often too difficult to answer basic questions such as:
- What topics are available within my business domain?
- Who is publishing and consuming events within this business domain and what data is exchanged?
- Do we already have something similar that we might reuse instead of defining topics/events from scratch?
- What are the best practices and conventions we might follow and how do we guarantee compliance?
AsyncAPI to Standardize Contract Definition
It is important to establish a common, simple, and precise language that allows us to describe all the aspects required to implement an EDA.
This is where AsyncAPI comes in: it is an open-source initiative that seeks to improve the current state of EDA by taking into account all those requirements and defining a common approach agnostic of technology.
The long-term goal is to make working with EDAs as easy as it is to work with REST APIs, including documentation, code generation, discovery, event management, etc.
EDA Governance Model
While AsyncAPI seems like a good choice, it’s not enough; we need to store these API specifications in an API Catalog. To achieve this, we will automate the execution of rulesets and data validations.
Let’s make use of Anypoint Platform and Confluent Kafka to define a governance model based on 3 phases:
Design Phase
At the time of writing, Design Center and Anypoint Exchange support AsyncAPI 2.0, except Avro schema.
Therefore, we can design AsyncAPI specifications using Design Center through a visual and guided approach, providing support and validations to simplify API definition. We will model the different events using RAML Data Types or JSON Schema.
This includes the definition of libraries to facilitate the reuse of common data structures between different contracts.
At the same time, API Governance will provide us with automatic validations as we design the API, based on the rulesets that have been previously selected.
There are out-of-the-box rules for AsyncAPI; however, it is also possible to define our own custom rules. As soon as we include them as exchange dependencies we will get feedback about warnings and errors, like in the example above.
Validation Phase
Once we have designed the API specification, we will request its approval and implementation by opening a ticket in the ITSM tool used in the company, such as Jira.
The contract is published to Exchange and a link to it is attached to the ticket. In this way, the corresponding IT team will be able to review the contract, confirm the level of compliance based on the API Governance rules, and carry out other types of manual validations to guarantee the company’s best practices.
If necessary, adjustments to the API design can be requested. Once everything is correct, we will implement the contract using the technology that applies in each case (remember, AsyncAPI is agnostic of technology).
Implementation Phase
Finally, we are ready to implement the AsyncAPI contract and, in this example, we have selected Confluent Kafka as our runtime.
The same approach and steps can be applied regardless of the technology to implement the contract. If it’s possible, I suggest you make use of Confluent Kafka, Amazon MSK (Managed Streaming for Apache Kafka), or any other similar product that provides a schema registry.
The schema registry stores schema definitions and is used to perform data validations on runtime, preventing wrong use of the topics when a producer tries to publish an event with an invalid data structure, and also helps the producers and consumers to understand how to serialize and deserialize events based on the schema definition.
The last characteristic is very useful when we work with Avro schemas, but taking into account that Design Center and Exchange don’t support Avro at this time, we are going to establish JSON Schema as the mandatory approach for the definition of data types.
Now, we will be able to import those JSON Schemas directly into the Confluent Schema Registry, without any transformation or manual tasks, avoiding possible errors during the process.
As a result, we will have registered the different topics and associated schemes in Confluent Kafka, so we can enable access to these topics to applications that require it.
Summary
We have seen how we can make use of the AsyncAPI specification together with Anypoint Platform and Confluent Kafka to define a governance model that provides the following benefits:
- Standardization of the definition of asynchronous APIs
- Automation of the use of quality rules
- Provides visibility into compliance with standards and best practices
- Offers a centralized view of the company’s integration strategy, storing the definition of synchronous and asynchronous APIs in a unified catalog that acts as a single source of truth
- Facilitates the discovery and reuse of assets
- Schema validation to avoid invalid data structures
Now we are ready to begin its implementation, for which it will also be advisable to review our C4E model to guarantee that we are prepared to respond to these new needs.
