How to select an Integration Platform for your Business — Part III
Let’s select the right integration platform for your business
This article is part III of a series of articles written on the topic. You can find the first 2 articles in the below links.
How to make your Integration platform cloud-native?
The integration platform that you choose should be future-proof and cloud-native or cloud-ready. The layman’s definition of the word cloud-native is “to reap the benefits of the cloud”. At a high-level, cloud computing offers you the capabilities like
- High availability — available across the globe
- Elasticity — can scale up and down as and when necessary
- Scalability — expand to global scales
- Cost savings — pay as you go
There are many other aspects, but we can consider these as the basic advantages of the cloud. Let’s see how we can align our integration platform to reap the benefits of the cloud. There is an entirely different set of advantages offered by container platforms and microservices architecture which is not mentioned here. Once you have the platform designed for the cloud, you can consider gaining those advantages also through the same.
One of the fundamental aspects of scalability and maintainability is modular architecture. If you have all the functionality baked into a single monolithic application, scaling becomes much harder. The concepts like microservices architecture have come up to address the problems which are surfaced with this monolithic architecture. Once the functional components are divided into compatible yet independent modules, the deployment becomes much more flexible. Let’s discuss how to define a cloud-native architecture for your integration platform.
The above architecture is an extension of the API-led architecture that we discussed in earlier sections. We have introduced a couple of new components to this platform so that we cover the modern requirements like event-driven architecture and analytics. Let’s discuss the components in this diagram in detail below.
- Messaging System — This component is required when there is a need to process a huge amount of asynchronous data points (events) coming into the system using an event-driven architecture. Apache Kafka and NATS are 2 of the best technologies available to fulfill this requirement. Depending on the levels of guarantees and the performance required for event handling, you can choose either Kafka or NATS.
- On-premise systems — These are the existing systems that are used in the enterprise which may have built with various protocols, standards, messaging formats. It is impossible to replace the capabilities of these systems with modern architectures like microservices or containers. Instead, these systems need to be integrated with the newly built architecture.
- Event processing system — This component is used to process the events received by the messaging system in real-time or in batch mode. This component can be used to transform, correlate, summarize, and store these events so that these results can be used for business operations as well as to improve customer experience. The processed results can be sent back to the messaging system so that other applications can receive these results.
- System APIs (Microservices) — There can be many situations where existing applications and systems cannot fulfill the ever-increasing consumer demands and business requirements. The microservices style of development is better suited for this sort of service development.
- Process APIs (Integration platform) — When there is a plethora of applications and systems that need to work together, the integration platform is the glue that connects various systems. Instead of using a traditional, monolithic integration layer, using a microservices style of a lightweight integration platform is recommended. The above figure depicts the usage of “micro integration” runtimes that are designed to support container-based deployments.
- Experience APIs (API platform) — Once the services are built as microservices or integration services, these services need to be exposed to various consumer channels. The API gateway acts as an intelligent gatekeeper that allows only the authorized requests while listening to all the requests. As depicted in the above figure, using a “Micro gateway” is beneficial in a container or cloud-native deployment.
- Developer portal — Providing a developer portal where external parties can build their own applications and products on top of the business APIs exposed through the system is critical in expanding the business to new heights.
- Analytics platform — Business Intelligence or Business Analytics is becoming ever so important with the amount of data collected in enterprise systems as well as the type of competition in the market. An analytics platform with proper tools and minds can really take you to the next level of your business and keep your competition at bay.
- Identity and Access Management platform — This is where the centralized management of authentication, authorization, provisioning, SSO, MFA happens. There are distributed alternatives like Open Policy Agent (OPA) that exists for microservices and API security. But not all the applications and systems support those alternative approaches.
The selection of the underlying infrastructure layer is totally up to the respective organization. But it is the role of the architect to design the solution so that it can be deployed in any infrastructure without much issue. The above architecture supports well with any of the below-mentioned infrastructure choices.
- On-premise (physical/VM)
- IaaS (VM based, AWS EC2, Azure, GCP)
- Containerized (AWS ECS, GCP, Azure)
- Kubernetes (EKS, GKE, AKS)
Deployment model
Now we have a better understanding of what functional and non-functional aspects need to be considered when selecting an integration platform at a higher level. The next thing that we should consider is the deployment model of the platform. This is as critical as any other point we mentioned so far. There are 3 main deployment models available for integration platforms.
- Self-managed deployment (on-premise, on-IaaS)
- Vendor-managed deployment (SaaS, private-IaaS)
- Hybrid deployment (vendor managed + self-managed)
Let’s discuss in detail what these deployment models offer to the users.
Self-managed deployment
This is the traditional on-premise deployment model supported by most of the integration (ESB) vendors from the past. This means that the user needs to manage the deployment of the integration platform by themselves with support from the vendor. The vendors provide support through ticketing systems like JIRA with different SLAs for different types of issues (incident, queries, development, etc.). With the popularity of IaaS providers like AWS, Azure, and GCP, more and more people started deploying these products on those platforms. Even in that sort of deployment on a cloud platform is considered self-managed if the user is managing the deployment-related activities like patching, updating, upgrading, and monitoring. This option is suitable for organizations having dedicated IT teams with enough resources to manage these deployments. With the usage of proper automation and monitoring tools, most of these management activities can be offloaded to systems. But that requires a significant effort to build such an infrastructure. One advantage of this approach is that the system has the highest level of flexibility in terms of customizations that are required for the specific customer.
Vendor-managed deployment
This is the option with the least management overhead. In this option, the vendor who offers the product takes care of the management of the deployment. In most cases, these solutions are provided as SaaS (Software as a Service) or PaaS (Platform as a Service) on the cloud. The users have to work with the solution architects from the vendor to identify the components and their respective sizes required for the integration project. Then the vendor will provision the required services on their cloud platform and give the users with access to those services. In most of the vendors, these services are running as multi-tenant, shared infrastructure with each user getting logically separated deployment. These solutions are commonly known as integration platform as a service or iPaaS.
One drawback with this approach is that the flexibility in terms of customization is very since this is a shared infrastructure. As a solution to that problem, some vendors offer the option of providing a fully-managed, private cloud deployment for users who are willing to pay some extra bucks. The deployment will be a dedicated one for the user on a given cloud platform (IaaS) and the vendor will take care of the maintenance and management aspects.
Hybrid deployment
This deployment model is designed to provide the best of both worlds in terms of flexibility and the management of the infrastructure. In this model, the vendor is managing the “control plane” or “management plane” related components that do not need to stay close to the user's systems. As an example, the API gateway is a component that needs to sit close to the process APIs and system APIs but the developer portal does not need to do the same. In such a case, the API gateway will be deployed in a customer-managed infrastructure (on-premise, IaaS) while the developer portal is deployed in a vendor-managed cloud environment. This model comes with a certain set of challenges as well. Integration from vendor deployment to user deployment needs to be set up in a proper manner and given the platform resides in 2 places, debugging issues can be difficult at certain times. But those challenges come with the price of flexibility and performance (latency) gains that users could achieve with this approach.
Each organization has its preferred ways of managing the IT systems and based on that preference, users can select a platform that supports that particular deployment model. One thing to consider here is that, if the platform is designed in a modular manner where functional components can be deployed independently, any of the mentioned deployment models is possible and future changes of directions can be accommodated.
Developer Experience
The next big thing that we should consider when selecting an integration platform is the developer experience with the platform. Depending on the level of exposure to the technology, the users (who build integrations) of an integration platform can be divided into 3 categories.
- Developers (programmers) — This is the category of users who are most comfortable with writing code and building integrations with programming languages. They will be comfortable with platforms that provide enough flexibility to extend the functionality in case they cannot find an OOTB feature to support a certain use case.
- Citizen integrators — This is the type of user who comes with a better understanding of the specific business domain (e.g. financial, retail, healthcare) but not always has the same level of expertise in writing code as the developers. They would prefer tools that provide higher-level abstractions like UI based flow designers.
- Integration specialists — These are the users with experience in implementing integration projects using various tools and technologies and comfortable with both writing code as well as DSL or UI based approaches. Some of them come with specific domain knowledge as well based on their previous experiences but not as much as citizen integrators.
It is critical to select a platform that aligns with the level of experience and the skill set that is available in the organization or has plans to hire. In most cases, we could find engineers who can wear different hats based on the need of the company. But it is good to select a platform that most of your engineers are comfortable with than asking them to adapt to the new platform.
Summary
In this article, we covered the fundamental requirements of an integration platform and then moved to more modern requirements that have appeared with the advancements in the enterprise technology domain. We used solution architecture diagrams to explain the overall architecture of an enterprise IT platform that has many other components in addition to the integration platform that we are going to choose. We identified various functional requirements at each component level. If you are in the process of selecting an integration platform for your enterprise, you should look at the most related functional requirements and architecture considerations mentioned in the article.
Given below is a list of leading vendors in the enterprise integration technology market.
This concludes the series of articles written on the topic of “how to select an integration platform for your business”.
