“Permission to ByPass Your ESB!” App Integration is your next big Monolith.
Integration, SOA, ESB’s, API’s, Blah, Blah, Whirlwind Vague History
Application Integration has been around for quite a while. It has evolved over the years. The initial integration platforms were built around “Messaging”, new applications would integrate with systems through MQ Queues. Initial Orchestration platforms were built around message flows. In a parallel path, Remote Procedure Calls like ONC-RPC emerged as another form of integration. This grew into distributed Objects like CORBA, COM, and eventually EJB. Eventually these platforms were deployed together into higher level integration servers built on top of Java Middleware based on Java EE driving centralized deployments. XML formats through web services became a dominant form of integration.
Reuse through web service calls (aka SOA) began to rule. That evolved into the need to manage them from a single location and thus the rise of Enterprise Service Buses (ESB). The need to “govern” through having a single “Platform and Team” where Web Service Calls go through became the main pattern for Enterprises. Organizations built whole teams around this pattern. These teams built “canonical data models” that represented (or attempted to anyway) a single data model for a business concept. It contained every data element across all different backend systems and thus these things grew as giant data models, supporting tons of applications.
Rise of API’s
- Back End For Front End (BFF): https://samnewman.io/patterns/architectural/bff/
- Interaction API, Experience API, etc……
One of the big API trends was to create an “API economy” that allowed exposing API’s to 3rd party partners.
Many centralized teams tried to fight back control as ESB’s began to also include API Platforms under the mantra of “API Economy.”
Now API Economy is something that exist in “certain” areas. There is a business model around exposing API’s to other businesses, and so the ability to create catalog of services is important “if I am really exposing API’s to other organizations that actually pay me”. However, within your own organization, if you are building your “own” front ends, the overhead of treating your own organization like external API consumers will slow you down. “Creating a charge back model within organizational walls to mimic an external API Economy suggests a bigger siloed organizational issue.”
As platforms evolved their ESB’s began to include API Platforms, they bundled all this functionality into a single platform under the the banner of “Integration”. As trends like microservices, data platforms, and Cloud furthered expanded, these platforms to handle integrating cloud data, on-prem systems, and so forth.
The Classic Integration Platform
The Classic Integration then evolved to represent the list of items below:
- Data and Logic Mappers (usually done visually) that have inputs and outputs.
- Some orchestration level visual tools to represent the ability to combine calls from several sources.
- Connectors to backend systems, some protocol based (http, messaging), and some data based (CICS, SAP). Data based connectors could introspect into some types of systems to create models that fit into tools that my data mapper could understand.
- Some security level mapping trying to abstract different security models.
- An API Gateway that provides advanced routing. This is a point of governance where i can create “rules” that allow not just scaling aspects, but selective routing based on consumers.
- Centralized place for metering and metrics of API Calls. This is perhaps the centerpiece of “governance.” This provides a single place of visibility.
- A messaging backbone to provide async behavior and attributes similar to “API’s” and expanded use cases like “publish and subscribe.”
Cloud Native, Microservices, DevOps, and Culture Change
Microservice development has lead to new models of application development where teams are broken into squads that are self sufficient. This way of development is often referred to as the Spotify Method. Rather than having separate Design teams, Development teams, build teams, operations teams, etc…, or splitting teams by technical layer, Spotify has a Squad that contains all the roles needed to build a microservice end-to-end. The Spotify model accounts for best practice sharing through the creation of chapters and guilds.
Many enterprises are adopting this style of development more and more, as it breaks down barriers, drives features into production faster. Organizations like my employer IBM, have created groups that help organizations shift into this paradigm (along with others such as Design Thinking, Extreme Programming, Lean Startup, Extreme Automation, etc….). At IBM, this is the Garage. Other organizations provide some level of culture transformation as well because it is needed to be successful with Cloud Adoption or Application Modernization.
Even though microservices in many ways resemble distributed objects, the emphasis is on Culture, DevOps, Distributed Deployments, and team structure.
Containers, Kubernetes, and the Service Mesh
This article is not about microservices or Cloud Native Architecture. There are too many articles on those topics. Cloud Native Architectures (12 factor apps) are now being implemented on premises, through Private Clouds facilitated by Kubernetes/Container based platforms like OpenShift. I am not doing a deep dive on this topic however, Kubernetes based platforms really provide some key functions that are of interest to integration specialist:
- Routing capabilities between containers.
- Routing to and from the container platform
- A management stack for metering, logging, and metrics for all things running in containers.
- Many kubernetes platforms provide the notion of a Service Mesh (like Istio). This provides a higher level ability to do advanced routing, gateways, advanced security attached to that routing, and rules.
- Related to the container platform are the implied DevOps tools that are needed to facilitate de-centralized deployment and scaling.
- Related to the kubernetes platforms are artifact registries that facilitate sharing of libraries, code, or docker images. Note, this is important because it opens up a “More flexible” reuse model that removes the version dependency issues that exist with a centralized service call. These registries usually integrate directly with Language build tools like Maven, npm, gradle, docker build, and others. Popular examples are Artifactory and Nexus.
What is Integration Today ?
As I work with these Enterprises today to adopt Cloud, App Modernization, Cloud Native, and Garage (culture agent), I am finding there is one role being left behind, and this is these “Shared Service Teams” or “Integration Team” that have built an Integration Platform for their backend services. On many clients, with a fast moving digital front end, I am seeing this layer being the slowest link in the chain.
Much like we began breaking up Monolithic middleware (or at least containerizing them), I believe that this centralized integration team is the next wave of modernization that is needed. Integration is more important than ever, because now we need to integrate not only across backend legacy systems, but new Cloud Applications running across various SaaS and Cloud Providers.
What are the aspects of a modern integration platform:
- The ability to create integration logic with both “low code” and modern “high code patterns”. We can no longer make the assumption that drag and drop development is faster than coding. New technologies like “GraphQL” provide a very efficient way in code to query against API’s and create new formats. As more SaaS providers create modern API’s, this will take over as the new integration language.
- Decentralized deployment of integrations is also important. The “Platform” is no longer an integration server, the platform is Kubernetes. The ability to deploy integration logic as a container is important. It makes integration feel like the rest of my development effort. This is often referred to as “Agile Integration.”
- Metering, metrics, routing, logging, and gateways now belong to the Kubernetes platform along with the Service Mesh. All my code (not just front door API’s) can benefit from modern techniques of centralizing these rules and the ability to configure it with Infrastructure as code makes it even easier to automate. This is where integration STOPS becoming the slow poke.
- API Best Practices still apply, but do not require a full API Management platform to achieve. Developers should use Swagger, REST best practices, and document endpoints and data formats.
- Externalize API catalogs should be for external consumers, and NOT for your own front end digital developers. The goal is to bring teams together.
- API Catalogs and Cloud Market places are becoming the same thing over time.
- Use of Kubernetes integration into identity for API keys and Cloud Services are needed for security. Security integration is no longer in the domain of integration platforms. Security services like encryption, and others should be available in general.
- Reuse can be achieved through various techniques, and a centralized API call is the “Least” flexible: (Git for copy paste programming, Git Clone for code reuse, API documentation based on something like Swagger or OpenAPI for API reuse, artifact repositories for bundling of reusable libraries and images, and Cloud API’s)
- Event Streaming and Messaging run as a backbone within the Kubernetes environment and available to all microservices.
Many of these concepts are described under the umbrella of Agile Integration. Here are some great resources on Agile Integration:
We're losing time and money with systems, apps, and technologies that don't talk to each other. How do we integrate all…
Organizational leaders who use an agile integration strategy can empower their teams to build and support the volume of…
Bring your plan to the IBM Garage.
Are you ready to learn more about working with the IBM Garage? We’re here to help. Contact us today to schedule time to speak with a Garage expert about your next big idea. Learn about our IBM Garage Method, the design, development and startup communities we work in, and the deep expertise and capabilities we bring to the table.
Schedule a no-charge visit with the IBM Garage.