Why “API Led Connectivity” can fail?
“API-Led connectivity” is the mantra you keep hearing over and over again — especially if you are into building integrations with API led integration platforms like Mulesoft. Allure of the mantra is that it claims to connect and expose organization assets and delivers integrations at speed and enable re-use. But in real world implementations “API Led Connectivity” fails to live up to the effectiveness of API Led Connectivity pattern— and is failing miserably.
Huh? What’s the difference between API Led Connectivity pattern and “API Led Connectivity” (notice the double quote!!) mantra?
What is API Led Connectivity pattern?
In simplistic terms, API led connectivity pattern is a way to classify and build the different types of APIs used in an enterprise — while specifying the high level functionality they are expected to implement. There are three main categories (or layers) of APIs in the API led connectivity pattern:
- System APIs
- Process APIs
- Experience APIs
System APIs are APIs which expose the underlying back end systems while insulating caller from changes to underlying assets.
Process APIs are APIs which create business value by working with single or multiple systems — typically done using one or more system APIs.
Experience APIs are APIs which reconfigure data for intended audience — tailored to the data format and requirements of the target system — typically done by calling one or more process APIs.
Advantage of the API Led connectivity pattern are multi fold like:
- Determine complexity and help in effort estimation
- Increased re-usability of APIs (only one API per the intended usage)
- Clearly outline the key life cycle elements of the API like Owner, frequency of change
- Identify operational and monitoring requirements
All sounds good — and it even makes sense. Then what’s the problem?
Problem 1: “API Led Connectivity” Mantra
API Led connectivity pattern has been adopted so well in the industry that it has become a mantra — “API Led Connectivity” (note the quotes for the mantra) with lots of unintended consequences!!
Let’s take an example. Imagine you need to expose order details from a database (or any back end system for that matter) as an API to be consumed by Salesforce (or any target system).
Typical “API Led connectivity” design for this use case would be:
- Create a system API to expose the database as an API
- Create a process API to call the system API — even if the process API is not calling any other system API or doing additional value addition
- Create an experience API to call the process API — even if the data format of experience API and process API are same
Hold on a second! That doesn’t seem right! And that doesn’t seem to be what the API led connectivity pattern seemed to be talking about as well — since no value creation is happening at process layer or experience layer. Then why is the API being designed and built this way?
Reason for this design seems simple:
- “API Led Connectivity” is understood to mean three layers and any deviation from this is considered taboo!
- Some business stakeholders would consider building without all three layers (whether the situation requires it or not) to be wrong design/implementation by the team!
- Some “idealistic” architects would like to keep the design flexible by adding additional layer for unintended requirements that might pop up in the future — whether there are requirements for that level of agility or not!
- Security concerns is another reason cited for creation of an additional layer without any value — even with the understanding that all APIs are by default built with security in mind, so no other layer just for security needs to be added in a solution!
Solution: Stop adding unnecessary layers if there is no need for it. You don’t have to build three layers for every API if you don’t have the need for it! And there are lots of ways to keep the implementation flexible for future changes without introducing layers in the design — explore options instead of following blindly!
Problem 2: Hidden Cost of “API Led Connectivity”
Why are we looking at cost when talking about the right way to implement? Would the cost mean just run time cost for the applications?
Cost for implementation of “API Led Connectivity” will include:
- Cost for design, implementation, testing and promotion per API (3 layers would possibly mean 3 API applications if implementing one API per application!)
- Run cost for running application (for example, see the image below for a sample comparison per layer using Mulesoft)
- Performance cost (SLA impact) of going through multiple layers for the caller
- Operational / support cost for monitoring and supporting multiple applications (health monitoring, log checks, insights, alerts, etc)
- Maintenance cost for bug fixes, upgrades and other activities on the application
Solution: Occam’s Razor states “the simplest solution is almost always the best”. Always consider hidden cost of design to understand whether its the most optimal way to solve a problem or not.
Problem 3: Wrong understanding of API Caller
In the example above, let’s assume from the above use case that there are 10 order fields captured in the database and only 2 of the fields are actually required by Salesforce (caller).
“API Led Connectivity” way would be:
- Create a system API to expose all 10 order fields from the database (hopefully this is how it’s built and not for just 2 fields which was in the project requirement!!)
- Create a process API (or an experience API or both!!) which will just take 2 fields requested by Salesforce and provides to the caller
What’s the rationale?
- Salesforce asked for 2 fields and system API had 10 fields, so we need another API to condense the response
- For future requirement if Salesforce wanted a third field, easy to modify the process/experience API
Solution: Before designing anything, check if the caller can ignore the rest of the 8 fields!! Most probably the caller would be able to ignore the unnecessary fields (if not readily, at least with a little convincing) — and the layer which was designed for this requirement might vanish completely!! In case the caller is not ready to ignore the unnecessary fields, implement the field selector feature in system API (in fact all APIs — no matter the type) — which should be a fundamental feature for every API anyway!
Problem 4: Wrong understanding of data format
Let’s assume that the order details from database is exposed as a JSON based API. Salesforce requires XML input. How would you handle this?
“API Led Connectivity” way would be:
- Create a system API to expose the order details as JSON based API
- Create a process API (or an experience API or both!!) which will convert JSON to XML
What’s the rationale?
- Salesforce asked for XML whereas system API is built true to JSON based REST principles, so we need another API to convert data format
Solution: APIs are not restricted to always being in JSON data format only! For most common and frequently used conversions within an organization (like JSON and XML), the feature for conversion should be provided by every API itself. In case the requirement is for a custom format as well, if the requirement is for all the enterprise APIs to support the custom format, then build it in all APIs and not as experience API per every API.
Final thoughts
Reason “API Led Connectivity” is failing is not due to the pattern — but from not following the pattern after understanding what it stands for. API Led Connectivity pattern definitely helps us to expose and connect organization assets with speed and agility. But true effectiveness of the pattern comes from understanding the design ethos of it and not from blindly following the marketing mantra.
Thanks for reading! I would love to hear your suggestions and comments!
