The fallacies of integration and the way forward
In certain periods of time, it rains a lot in the subcontinent. When my brother and I were little kids we used to run around in the rain. We loved feeling the raindrops that fall from the sky. That is until our mother forced us to get inside the house. She believed getting wet in the rain makes us sick. It is true that getting wet could make you sick but getting wet itself cannot make you sick. Our skin evolved for millions of years and it is very good at protecting us from water. What actually happens is rainy conditions drop the temperature in the environment making it ideal for pathogens such as bacteria, viruses and that kind of stuff. On the other hand, getting wet in the rain drops our body temperature as well. Therefore, there is a high probability that pathogens get into your body and then thrive in it, which actually makes you sick. So the thing is what you see is not actually what happens behind the scene. It is a fallacy.
The same applies for the story of complex integration. When you talk about integration the first thing that comes to mind is Enterprise Service Bus (ESB). So let’s start with that.
Drag and drop editor, DSL and ESB make us productive
I was part of the fastest open source ESB in the world. It was the best selling product in the company. When I joined the team the product was already five years old give or take. It had already had a lot of big customers under its belt. I mean really big customers.
I was excited when I heard the news that another huge customer in the US wants to use our ESB for their integration. The team decided to send me to a customer site for three months to help complete the project as a consultant. Now, I was even more excited. For the first time I am playing the product user role instead of the product developer role.
It went well at the start and the original requirements were implemented at ease. But then overtime came the unforeseen never ending train of change request which made integration bit by bit complex. This actually turned my excitement into a frustration as the Domain Specific Language (DSL) and Integrated Development Environment (IDE) started falling short. They lose their advantage drastically as the integration evolves into something complex. The tool I have been given for complex integrations was inadequate and different and yet I was expected to be productive in what I am doing.
Complex integrations also pushed the DSL to forefront and the drag and drop editor to the background. Because it is easier to read the DSL instead of going through a never ending set of arrows and boxes. This made my frustration even worse as I am programmer who is good at general purpose languages but now I have to work with a DSL. DSL’s are very different from general purpose programming languages and in fact they are very different from one another as well. As a programmer I didn’t really enjoy dealing with DSLs. It was like asking someone who is good at eating food with forks to suddenly eat food with chopsticks.
Why was it exactly inadequate and different? programmers have to learn a DSL instead of using a regular programming language, debugging is inferior compared to a regular programming language tools, DSL often falls short forcing the developers to use different things such as JSON schemas, XML schemas, scripting languages and of course the fully fledged programming language behind the DSL, reading the DSL and evolving the integration is not easy, tooling suggestions are poor and primitive, code organisation is different and inferior to a regular programming language, and error handling is cryptic.
Our customer managed thanks to the DSL’s ability to invoke the underlying general purpose programming language. As a matter of fact, this would have been the case even if they had went with some other vendor. IMO, they actually made the best choice given the tools they had at that time.
Interesting part is these tools struggles to meet complex integration needs and yet they are still considered as competitive and relevant, which is a common fallacy. I don’t have facts but I believe it is because drag and drop does look very appealing when presented with a pre-built demo. It just has this tendency to fool people. We all like to think that we make objective decisions after carefully weighing facts but in reality your subconscious mind already makes decisions for you even without your conscious mind knowing about it. After all, your brain is an energy hog and nobody wants to burn out energy.
Also, there could be a bit of politics at play as usual.
If you really think about it, these tools are counterproductive when the integration is complex and your team is full of decent programmers.
Don’t get me wrong, DSL is good for simple integrations but not for complex ones. On the other side, the ESB itself is becoming obsolete as the industry is moving towards more agile practices which are based on independent development, deployment and scaling. Something that is not possible when you deploy things into a shared ESB. This shift was a big deal actually and it was clear for me and our team that the product we loved and passionately built is starting to see its end of life. It was a bitter pill to swallow but the sun finally has started to set, making room for products that can do lean and clean integration.
Some programming language, a framework and bunch of libraries is all we need
After a few years I just heard that the customer we had, decided to move away from bulky centralised ESBs. It was sad news but not a surprise. They just wanted something decentralised and lean. Something allowed them to independently develop, deploy and scale. This was the trend in the industry as well. But what choices did they have to be productive apart from ESBs? Nothing! This gave room for some unusual products in the world of integration. They managed to take out the darkness left by sunsetting the ESB but not by much. If you ask me, leading poor developers to the dark age of integration.
As an alternative to the above requirement, people started using frameworks such as Spring boot, Quarkus, Micronaut, etc. They are just frameworks built on top of some Object Oriented Programming (OOP) language. They are decentralised and much leaner than an ESB runtime fitting in nicely with the story of independent development, deployment and scaling. However, they are NOT specialised in integration. It is lean but it is not clean.
Let me explain why. At a fundamental level this is how they look. To begin with they are usually built on top of some OOP language. These languages are great for building large applications. They are very good at breaking down large applications into smaller components by creating boundaries. These boundaries are usually defined using encapsulation, interfaces and that kind of stuff. They see everything as objects which usually have state and some sort of behaviour.
But the thing is, the boundaries in the context of integration are network boundaries and each component usually has its own isolated process. This means a lot of things. To begin with these network boundaries accept pure data and send back pure data NOT objects. This data can be in JSON, XML, CSV, YAML and if you are old enough it could even be SOAP. Data can have optional values, default values, nil values, additional fields trying to map them to objects results in unwanted boilerplate code with annotation.
APIs of the component are now not exposed as a simple function but as REST, GraphQL, gRPC, WebSocket, WebSub and list goes on. Trying to fit them into functions again results in a lot of annotations besides there is not one technology who is good at all of them either.
Then comes to the matter of describing these APIs, network APIs cannot be described using traditional Java like interfaces, instead you need to use something like OpenAPI, RAML, GraphQL schema, AsyncAPI, etc. None of the OOP abstractions can describe the above interfaces without settling for annotation.
No, I am not done yet. Traditionally, components were invoked in the same process, usually using the same thread. But in the case of network APIs, an unlimited number of concurrent clients could invoke the component APIs. This means dealing with concurrency can’t be an afterthought. Current programming languages constructs are not designed in a world like this. During my ten plus years of experience I have seen plenty of concurrency issues, spoiling my weekends.
Also, error handling is not the same, the network is inherently unreliable unlike in-memory invocations, therefore, error handling needs to be in the forefront if you want a reliable API. You must treat errors as expected but undesired results. Errors are now part of your program just like normal values.
Finally, the usual table joins SQL query don’t work anymore, as these tables are distributed across different services. You need something new to talk to those services, get the data, join, filter and aggregate them.
So it is clear that tool that developers are using is built for a different purpose. All these misalignment makes developers work hard to produce little. And yet somehow some developers believe what they have now is the best solution for their problem, which is a fallacy.
Remember, it is one thing to just do integration, it is another thing do to integration that makes development, deployment, operation and maintenance smooth.
What we need is tools that can truly fit into integration needs.
What if there is a better alternative?
I witnessed a product going from its best days to worst days. Every product has a life cycle and products reach the end of their usefulness in the markets they compete in. That is when our team decided to come up with something new and I am glad that I was part of it. I was lucky to see two ends of the software product lifecycle. I saw the end of life of a product as well as the birth of a new product.
Are integration developers stuck with the above fallacies forever? Well, there is an alternative which can truly make you productive: Ballerina Swan Lake. It is a programming language designed from scratch, yes, from scratch to make integration lean and clean. It just inverts the experience you had with visual editors and ESBs, it brings programming to the forefront and moves the visual editor to the background. The visual editor is still there when you need it the most though.
As an integration developer you might be thinking, okay, you are saying I need to learn another programming language? Yeah. But unlike cryptic DSLs if you know any programming language basics you already know a lot of Ballerina Swan Lake.
Ballerina type system is designed to deal with network data that comes in and goes out as part of your network APIs, eliminating a lot of boilerplate code. Its concept of services allows you to design many different APIs such as REST, GraphQL, gRPC, WebSocket, WebSub, etc. These APIs can be designed and visualised using the visual diagram. Compiler guides you on writing concurrent programs. It detects unsafe data access and warns you. Unsafe services are not run concurrently. It has a vast array of connectors allowing you to connect to multiple systems. It has SQL like query expressions to join data from multiple endpoints and project it to the form you want. Data transformations can be done with the data mapper. It has a test framework designed for testing integration scenarios. Ballerina services has excellent CPU and memory utilisation and amazing startup times.
So, Ballerina Swan Lake is not only makes your integration lean but also clean. It truly gives developers an experience that makes development, deployment, operation and maintenance easy. This means you can do more with less while having fun. Finally, a smooth ride from the start to the end.
It is time to say goodbye to fallacies. It is time to give developers the power back to make integration great again. It is time to replace DSLs and ESBs with Ballerina Swan Lake.
If you are a non-developer, you have good news too. Sun setting the ESBs led to altered versions of DSLs, IDEs and runtimes, which is capable of independent development, deployment and scaling such as WSO2 Micro Integrator.
After years, finally, the integration landscape is complete, making everyone happy!
Ballerina is programming language by developers for developers. For more information see [Introductory] Integration with Ballerina.
