Why is the ESB still relevant?
The most common argument I come across these days is the future of the ESB (Enterprise Service Bus). For many years, the ESB has been the key to building a loosely coupled integration platform capable of converting any protocol and format. However, many companies are starting to invest in a service-mesh and event-streaming platform, in the look-out for an alternative to their integration needs.
When I was first pulled into this conversation, I was very confused as the purpose of the ESB is very different from that of a service-mesh or an event-streaming platform. Later, I realized that companies are looking to solve the problem of integration within their applications, and focus on sharing the data versus than transforming data. Before we look into the existential crisis, let’s take a minute to look back at ESB came into existence in the first place.
Until the 90s, most companies had their applications interacting with each other, almost like people in a parliamentary session. A lot of point to point interaction with very discipline. At some point, it was decided to bring in a moderator who can efficiently listen to everybody’s opinion and stitch them together in a way that each of them will understand. The communication was so easy as the transformation was efficient, irrespective of the message format — be it copybook or EDI formats to SOAP/XML or JSON. The Canonical Data Models standardized the flow of data that enabled the organisations to efficiently partner with third-party applications at will.
The ESB has solved all these problems; but how did it manage to become a problem by itself? Well, the answer is the implementation pattern, leading to strange decisions. We live in a world where Pant and Pandya are sent ahead of Dhoni in a World Cup semi-final. Instead of letting the applications do the work they are supposed to do, the architects and designers started to exploit the ESB and write their business logic into it. Not every ESB implementation had its CDM data model and in a few cases, their orchestrations were ineffective. This often resulted in changes to the eco-system becoming more tedious. From being a catalyst, the ESB ended up as a bottle-neck in those few cases.
The ever-changing or rather ever-evolving trends in technology, from SOAP services with CDM, ESB has moved on to REST APIs (thanks to kubernetes) but the importance of the data model is lost in this madness. In the name of micro-services, point-to-point integration with simple transformations being pushed on as ESB forgetting the very reason for implementing an ESB. In other words, instead of a moderator who enables effective communication, we got ourselves into the “Newshour Debate”. Remember, most of these ESB products come with licence costs. This combined with the shift of trend toward service-mesh, event streaming, fast data has become lethal for ESB.
Switching back to our primary argument, is this the end of ESB? The answer is not simple. It purely depends on what the organisation wants to do (or how they see themselves being positioned in future). For tech-first companies, it is an effective strategy to invest in technology and to stay on top of the trends. But tech-enabled companies that look up to their technology wing (who have to play with a very limited budget) to provide a stable and cost-effective solution to their needs would have to consider which technology would be the right fit for their problems rather than simply staying with the trend just because everybody else is doing so.
The mind-boggling question is: does keeping up with trends influence the cost-effectiveness of the solution?. To answer this, we must first look at how an enterprise would function without ESB.
Firstly, When all the applications in the enterprise follow an API-first architecture, the need for a mediation layer to support multiple data formats becomes less significant. Secondly, The complex transformation and the stitching of data are expected to be taken care at the application end rather than an ESB.
So the problem that remains is that of sharing the data effectively, which service-mesh and event-streaming platforms take care of.
Tech companies, as they move out of ESB, they have a clear roadmap of what needs to be done. Rather than changing their architecture overnight, they transform their applications one by one before letting the ESB go. Most tech companies that you see making their waves with the shift in the trend has already gone through this process (or mid-way through it).
If we consider the case of tech-enabled companies, especially the ones with a small purse for investments in technology, they have to rely on COTS (Commercial off-the-shelf) products to make their solutions quick and cost-effective. The problem with COTS is that they are not flexible. Every change costs. That’s where the ESB comes plays a role. From complex transformation to orchestrating and stitching the data together, the ESB becomes the central nervous system. When these companies want to move out of the traditional ESB approach, the one question they should ask themselves “are they ready for it yet”?
In most cases, the answer is NO. Reasons? Cost of developing in-house solutions against that of COTS, and the ability/skill to do so. And hence the better solution for these companies is to stick with ESB and give themselves the time for them to evolve beyond traditional ESB rather than to jump into the trend.
The ESB is an architecture; it is up to the architects to design implementation patterns that allow the ESB to do what it is intended to do rather than making it a “Newshour Debate”. With the proper architecture and disciplined implementation, I say the traditional ESB is here to stay. At least for the next decade if not more.
