Hybrid & Multi-Cloud Integration

We all know that the adoption of cloud services is increasing and many companies are starting to adopt a “Cloud first” strategy. As a result the majority of organizations will be living in a hybrid environment for quite some time. With the enterprises I have been working with over the last 10 years, this tends to be due to regulatory requirements, unique hardware & software requirements, and the sheer size of their existing application portfolio that needs to be moved to the Cloud.

I have seen (and assisted) enterprises, subscribe to multiple disparate SaaS applications, develop new cloud-native applications, and migrate on-premise applications to the Cloud.

While adopting cloud services facilitates innovation, I have seen enterprises being hindered in their ability to innovate using cloud applications due to a lack of integration.

Enterprises soon realize the need to develop a pragmatic and holistic approach to integrating cloud-based applications with each other and with on-premises applications.

Questions I ask my customers are:

• Are your competitors executing their digital strategy and integrating new cloud services faster than you? — Maybe you have a growing backlog of business initiatives that require integrating with cloud partners while your competitors seem to be delivering faster than you, its probably time to take advantage of new cloud integration architectures and technologies.
• Do you have a significant percentage of your customers coming to your website through mobile devices, are they getting consistent experience across different channels while at the same time encountering a mobile optimized experience or are they simply being frustration?
• In addition to your customers, what about your employees are they able to rapidly access all the applications and data they need within your organization, whether they deployed in the cloud or on-premises?
• And finally, looking into the future, as new integration channels are added to your infrastructure, is your current infrastructure too complex to support IoT and other initiatives?

A consistent and repeatable integration approach is needed to address cloud-to-cloud integration, cloud-to-on-premises integration, mobile applications, integration of devices to support internet of things capabilities, and business processes that leverage functionality from on-premises applications and cloud-based applications. I thought I would take some time to share some of my experiences, and best practices with assisting enterprises to architect and develop an integration approach in addressing their cloud integration needs.

To understand where an enterprise is today I collaborate with enterprises to answer the following questions

1. What is the best approach to integrate on-premises applications with SaaS?
2. What is the best approach to integrate SaaS with SaaS from one cloud vendor or across multiple cloud vendors?
3. What is the best approach to add mobility to existing applications whether on premises or cloud-based?
4. How can business processes be built to span on-premises applications and SaaS?
5. How can devices be integrated with applications, both on premises and cloud-based, to deliver internet of things capabilities?
6. Where should integration capabilities be deployed — on premises, in the cloud, both?
7. How can integration efforts be prioritized to deliver the most value in the least time?
8. What cloud services are needed to support these hybrid integration requirements?

Before we start let me level-set some terms from a “Cloud Integration” perspective. Specifically the terms, Multi-Cloud, Hybrid-Cloud, and Hybrid IT, as sometimes they are used interchangeable — For the purpose of this musing, I will use the terms in the follow manner. (You might not agree with these definitions but at least you will now what I am referring to)

• Multi-Cloud is a strategy of utilizing and combining cloud services from multiple public cloud providers.

• Hybrid-Cloud is a strategy of utilizing and combining cloud services from both a private cloud and public cloud providers.
• Hybrid-IT (as stated by Gartner) is the development, execution and governance of integration flows connecting any combination of on-premises and cloud based entities whether they are processes, services, applications, and data — whether they are within individual or across multiple applications. (Phew — that was a mouthful)

Identifying Patterns and Platforms

Patterns are a proven approach to solving integration problems (e.g. the Enterprise Integration Patterns created by Gregor Hohpe and Bobby Woolf). Patterns provide a means to capture and reuse successful approaches to solve recurring integration challenges. Patterns can also be combined in a myriad of ways to address more encompassing integration initiatives.

I have (and will continue to) document integrations for cloud to on premises and cloud to cloud services. These integration patterns differ from other integration patterns such as the Enterprise Integration Patterns in that these patterns are at a higher level of abstraction and focus on where the components are deployed (on premises, cloud, or both) and the interaction of the underlying integration capabilities.

Use Cases

Rather than having an ever growing list of patterns on which to trawl through I decided to adopt a Use case approach to narrow the number of patterns that might be applicable to a particular integration requirement. The diagram below illustrates a sample of some use cases that commonly apply to a holistic hybrid integration initiative.

Sample Integration Use Cases

1. SaaS to SaaS (Single or Multi-Cloud) — Due to Shadow IT or business pressure I have seen a lot of enterprises with multiple SaaS services that have been subscribed to without any future thought of their interaction and integration needs. Line of Business (LoB) organizations subscribe to what they feel is the best SaaS application for the job. As companies embrace more SaaS applications, the need to integrate the SaaS applications will increase. In most cases the SaaS applications will be from different cloud vendors.
2. SaaS to On-Premises Application — In many instances, SaaS is chosen to provide additional capabilities that are not provided by the existing on-premises applications. This then results in the need to integrate the SaaS application with existing on-premises applications.
3. Application in the Cloud to On-Premises Application — Enterprises that have lift and shifted a portion of their application portfolio soon realize the need for re-establishing existing integrations to the on-premises applications. This is also true when new cloud-native applications are developed and deployed in the cloud, more than likely these new applications will need to integrate with on-premises applications.
4. Application n the Cloud to SaaS/Application (single, multi-cloud) — As on-premises applications are re-hosted to PaaS or IaaS, and new cloud native applications are developed there will be a need to integrate with SaaS applications, which may be provided by a single or multiple cloud providers.
5. Application in the Cloud, Data Origination On Premises — A common issue that arises when moving an on-premises application to the cloud is the existence of shared data sources i.e. data sources that are shared by several on-premises applications. Moving all of the applications en masse to cloud is risky and impractical. Therefore access to the on-premises data needs to be provided to the re-hosted application. Initially the “data access gravity pull” is greater from the on-premise applications rather than say a single application in the cloud. This can obviously change in the future as more and more applications in the cloud need access to the same data store. If read-only data in the cloud it might make sense to replicate this data from On-Premises to the Cloud. Another common example can be due to regulatory or security policies that require the data to remain On-premises.
6. Mobility — Mobile devices have become ubiquitous, and the need to provide function-rich mobile applications has increased. Mobile applications do not have to contain the functionality; rather they expose functionality from existing applications which could be deployed on premises, in one or more clouds, or a combination.
7. Cloud Sense — Integrating devices into the hybrid IT environment requires that the events and alerts sent by the devices be captured, processed, and then an appropriate response be taken. PaaS can provide the capturing and processing of the events and alerts while the response could be provided by an application on premises.
8. Cloud Respond — As more IT assets move to cloud, the application(s) responsible for responding to events will likely be cloud-based. In this case the capturing and processing of events might be done both in PaaS and on premises while the response is cloud-based
9. Managed File Transfer — Unfortunately Managed File Transfers (MFT) still seems to be the most integration approached utilized on-premises today. In a hybrid IT environment the source and targets of these managed files can be on-premise and a cloud service (single or multi-cloud). The consolidation and distribution of files from multiple clouds to a single staging area or a fan out from a single source to multiple targets is a common need.

These use cases are not exhaustive but should give you an idea of the many integration requirements and nuances. How many of the use cases that apply to any particular enterprise depends on many factors including their existing IT environment, their adoption of SaaS, the desire to re-host existing applications, mobile strategy, and internet of things adoption, etc.

Platform Deployment Options

For each integration requirement, identifying the use case reduces that number of patterns that could be applied. Nonetheless, for each use cases there are multiple patterns that could be applied; therefore some analysis and trade-offs need to be done. For example, lets look at the first two use cases.

Use Case #1 (SaaS to SaaS)

When the SaaS applications are in different clouds, a decision needs to be made concerning the location of the integration capabilities. For instance Centralized or Federated?

Cloud to Cloud Integration

The centralized approach uses integration capabilities (e.g. Integration as a Service) in only one cloud so that cloud becomes the locus of all cloud-to-cloud integrations. This approach exhibits faster time to integration with the use of newer integration platform capabilities and an ever increasing adapter marketplace.

The federated approach has the integration capabilities hosted in multiple clouds. This can come about after an acquisition of a enterprise that utilizes different cloud vendors. Therefore it is common to see this deployment option as a short-term approach until one of the platforms are sunset. The federated approach can also be viable if there is an equal localized integration gravity pull within each cloud vendor environment. This approach does increase complexity compared to having only one platform, and has the potential for a performance impact due to the integration going through two platforms.

Use Case #2 (SaaS to On-Premises Application)

This use case has a similar high-level deployment option that needs to be decided upon.

Cloud to On-Premise Integration

The on-premises option uses existing on-premises integration capabilities to integrate cloud-based applications with on-premises applications. This is the most likely “initial” approach for enterprises, who wish to leverage an existing on-premises integration suite skill set and investment. These on-premise platforms may lack cloud adapters and are sub-optimal for cloud to cloud integrations.

The hybrid option uses integration capabilities both on premises and in the cloud. This is a common approach for Hybrid IT enterprises to adopt. It provides the flexibility to adopt a center of gravity approach to integration. But it does increase complexity compared to have only a single integration platform. In addition there are potential performance impacts due to integrations going through two platforms.

The cloud option uses an Integration as a Service Cloud Platform.This approach is a good fit for companies who insist on the ease of use, ease of maintenance, time to deployment, and are on a tight budget. It is appealing to small and mid-sized companies, as well as large enterprises with departmental application deployments. This is also a good longer term option for larger enterprises, once majority of applications are in the Cloud

There are pros and cons for each of the approaches, and which option is most appropriate again depends on many factors specific to each enterprise, and potentially, specific to each integration. It is important to note that the deployment options are not mutually exclusive. A company could choose to use different approaches for different integrations. Or, the approaches could be transitional architectures from the current state to the desired future state. For example, when a company has only one or a few SaaS applications, the on-premises option maybe the best approach. As more applications are cloud-based, the hybrid option may become the better choice. And finally, as the transition to cloud nears completion, the cloud option may be the desired approach.

Pattern Definition

Currently there are over 30 patterns which I have put into a downloadable poster

This poster provides a conceptual view for each pattern. The poster can help to identify which pattern(s) might be applicable to an integration requirement.

I am hoping later to have time to go in more depth on the interaction diagrams that supports this poster.

Good luck, Now Go Architect….