Understanding the modern enterprise integration requirements

Enterprise Application Integration (EAI) is a complex problem to solve and different software vendors have produced different types of software products like ESB, Application Server, Message Broker, API Gateways, Load Balancers, Proxy Servers and many other forms. These products have evolved from a monolithic, heavyweight, high-performing runtimes to lean, modularized, micro-runtimes. Microservices Architecture (MSA) is having a major impact on the way architects design their enterprise software systems. The requirements which were there 10 years ago has been drastically changed due to modern advancements of MSA, Containers, DevOps, Agility and mainly due to crazy customer demands.

If you came this way to understand about new EIP patterns, this is not the right place. In reality, the 65 EIP patterns which were introduced by Gregor Hohpe is still in action with few more additional patterns coming through. Refer the below link to understand the EIP patterns which you can still use with the modern architecture which is presented here.

http://www.enterpriseintegrationpatterns.com/patterns/messaging/

Another misconception is that “Integration” means just connecting different systems which communicates using heterogenous protocols and messaging formats. In practice, integration architects needs to understand the whole eco system of an enterprise software architecture.

Figure: Modern enterprise application integration requirements

The above figure depicts a typical integration architecture within a modern enterprise. It consists of the following layers. You many find these layers glued together in some scenarios, but overall, these functionalities are existed.

• Data layer — This is the layer in which your business data resides. It can be databases of type RDBMS, NoSQL or ERP/CRM systems like SAP, PeopleSoft or MS Dynamics or any proprietary systems which you have built.
• Service layer — This is where the MSA and the technologies like containers, docker, kubernetes are started to make an impact. With the concepts like Domain Driven Design (DDD) or “doing one task at best” type of approaches have made this layer more lean.
• Integration layer — This is where the traditional integration happens with service orchestrations, data transformations, protocol switching and all EIP patterns which are mentioned above play a major role at this layer
• API management layer — This has become an integral part of any integration project where it provides a clean abstraction for the business as well as consumers to act upon. The capabilities like authentications, authorization, caching, throttling and analytics are provided at this layer so that every integration or back end service does not need to worry about implementing it.
• Load balancing layer — This is the layer where network and threat protection alongside load balancing is implemented. External attacks like DOS, XML injections are prevented at this layer

It is quintessential to know about these layers if you are designing enterprise software system. But knowing them and the features at each layer is not enough to make decisions when you are actually doing the implementation. You need to understand the main requirements of each and every layer at an architectural level before deciding on specific technology or vendor/s.

Data layer

This is where the business critical data resides in your system. You should have the following main requirements covered with any of the technology/vendor you are going to select for the implementation.

• Consistency — The data you stored in this layer should be consistent. Any type of tampering of data needs to be avoided and necessary measures needs to be taken to prevent such attempts.
• Persistence at scale — Your system should be capable of storing events at a very higher rates without losing consistency. In the meantime, it should be available to read the persisted data.
• Security — This is a critical requirement of your data stores since this data derives the value for your business. Any unintended use or access needs to be prevented

Services layer

This layer consists of actual business logic and services which offers the information which is needed by most of the client systems. In SOA world, you may have few services with each service covering a particular area of functionality at a macro level. But in a MSA type of implementation, each service covers a unique functionality which is at a micro level. As an example, you may have a flight booking service at SOA world and a flight information service, flight schedule service and flight rate service as separate microservices in a MSA world. In any of the situations, you need to deploy these services indepedently so that they will not have an impact on each other when there is a failure. At this layer, the system should be capable of fulfilling the below main requirements

• Agility — This layer should be implemented in an agile manner with proper testing and automation strategies built around that. Since this layer provides the more up-to date business critical information, new services needs to be implemented more frequently when the business grows. Proper CI/CD processes and build pipelines needs to be integrated with the selected technology.
• Latency — Another important aspect of this layer is the latency. The results should be delivered with minimum latency since there are additional layers which adds up to the overall latency due to network interference. The services implemented at this layer should more clean and efficient to provide better latencies.
• Automation — The other important factor to consider here is the level of automation and the different levels of testing carried out before releases. Since this layer is at the heart of the enterprise system and the core of your business, any false data or service outages can cause loss of business.

Integration layer

This layer provides the mediation capabilities required for different systems to interact with each other. You can find tons of different integration products and technologies which provides the feature set which is required at this level. But not every technology/vendor is capable of fulfilling the below mentioned critical but future-proof requirements.

• Simplicity — Integration of services need not to be as hard as implementing business logic at services layer. You should have easy to use syntax and visual tooling to build these integrations without much fuss. In the meantime, it should provide the necessary building blocks to deal with integration requirements at production deployments. Concepts like data types, resiliency, containerization needs to be supported.
• Agility — This might have not been a requirement in the past where you had bulky ESBs, Message Brokers deployed for integration. But in a modern enterprise, your integration technology needs to be agile and capable of supporting frequent releases, automation and devOps integrations
• Stability — When you deal with heterogeneous systems, there is a high chance that the system might interact with lot of unexpected data types. This can cause stability issues in your integration layer. It is essential that your integration system should be capable of handling such scenarios without losing the stability
• Resiliency — Another aspect of integrating heterogeneous systems is that those system can fail at any moment and the integration layer should be able to withstand those failures without propagating those failures to upper layers. The technologies like Circuit Breaker and retry needs to be supported for resiliency.
• Performance — When it comes to performance at the integration layer, it should be capable of accepting concurrent requests at a higher rates while dispatching them at a lower rates to the backend services which are not that capable of handling concurrent connections. This capability is critical when you have systems which has different message processing rates.
• Caching — This is kind of nice to have capability at the integration layer given that it is much closer to the actual back end service. If the backend services layer is malfunctioning, integration layer can provide some sort of cached data in some scenarios.

API management layer

Once you have the backend services layer and the integration layer implemented, the system is ready to expose valuable information to external systems. If your system is dealing with internal systems, you might not need an api management layer. But in most of the pragmatic integration projects, api management layer is a core requirement. Instead of implementing common requirements at each and every services level, it is more efficient and scalable to add an api management layer. You will find dozens of API management vendors in the market which offers the same set of basic features required for traditional API management requirements. A more modern api management solution should be able to cater the below set of requirments.

• Authentication and Authorization — The system should be capable of authenticating the users who requests access to the services layer. Authorization can be an added feature to secure the services at a fine grained level. OAuth2 has become the defacto standard for access delegation and authentication. The system should be capable of handling OAuth2.
• Throttling — The system needs to be properly controlled so that users get a fair share of access across the board and if you are charging for usage, you need to apply more advanced throttling policies.
• Analytics — Monitoring and analyzing your API usage is critical to improve your business processes and drive new revenue. Business level analytics is more critical here alongside with system status monitoring.
• Availability — High availability is critical at this layer since users don’t want to see empty pages or internal server errors at any given time. By making this layer highly available, we can account for failures at upstream layers like integration and back end services through caching at API gateway.
• Resiliency — Another critical requirement at api management layer is the ability to withstand failures at upstream layers. It should not propagate the failures of those systems to other downstream layers like load balancers. This can be achieved by having mechanisms like circuit breaker, bulkhead, retry at this layer.
• Caching — Intelligent and adaptive caching technologies can be useful to provide uninterrupted service to the consumers. Caching helps to improve the overall performance of the system by reducing the load on the backend systems. When the back end services are not available, intelligent caching systems can still provide cached results to the consumers.
• Developer experience — If you are exposing your APIs to the developers to browse through the APIs and create their own applications, developer experience is another key requirements of the system. Sometimes developers may need to build their own APIs by composing existing APIs. This API composition capability is a nice to have requirement in most cases.

Load balancer/Proxy layer

Exposing your business functionality to public internet needs to be done very carefully given the number of data breaches and hacking activities happen around the world. That is why you need to have a proper load balancing or proxy layer to provide basic protection to your information system. Main requirements which needs to be fulfilled at this layer are

• Security — This layer should be able to handle basic security threats like DOS attacks, XML injection, CSS, CRLF and others. It should also be able to support SSL.
• Resiliency — At this layer, resiliency means that any failures in the upstream systems must not take this layer down. When the upstream systems are not available, it should be properly handled at this layer through techniques like circuit breaker, retry.
• Performance — This is the layer which will serve all the requests coming into the system. We can reduce the load on back end systems by using caching technologies at other layers. But this layer is not able to do that since it is at the forefront of the users. Having a good performance in terms of concurrency handling is essential at this layer.
• Caching — Having caching capabilities at this layer will eventually improve the overall system performance by reducing the load on the upstream systems. But the caching needs to be intelligent and adaptive since there are multiple layers of data invalidation happens at the upstream systems
• Load balancing — Obviously, this is one of the core capabilities required at this layer to handle the enormous load coming into the system.
• Availability — This is the layer where availability is most critical. Since there are no other ways to mimic the availability of this layer (similar to caching), having HA at this layer is a must.

As discussed in the above sections, having a layered architecture allows us to think deeply about those layers and their requirements. Also it makes the way to choose the best suited technology or vendor at each layer. There are technologies/vendors who are highly praised by analyst reports for each and every layer. It is the task of solution architects and CTOs to take the final decision.