A tale of two Queues and a Camel
This is a WIP story.
Lets start with the problem statement. I have been tasked to integrate a quite big system backend with the an other, a comparatively strategically bigger system. “Strategically” is the keyword here. And this system B is the strategic system for couple of reasons like:
- The focus of the company. To put it concisely, That was the premise with which system B won over system A to start with.
- Cost vs Benefit analysis. Would be good to explain this; a) System A and System B has quite different set of customer base. Unifying the backend gives us completely different customer base. Double the mileage in some sense. b) Would give the partner power of two also. Partners also would say why not? And may be get the efficiency overall.
I would start with quantifying that this problem is no mean problem by any means. And nobody said so. Hence it is a big diligent exercise and there could be different ways to solve this.
Now the nuances or the facets of the problem are required to clearly elaborate the problem. Hopefully with it we will reach describing the complete problem in one statement which is always a good sign of achieving the goal of clearly defining the problem.
- The two systems are separate in each and every sense technically. They have different a) Infrastructure, b) Technology stack, c) Different Data Models (with some commonality understandably), d) Different features (same, like some commonality here too), and e) Somewhat Different business processes also.
- As it has been mentioned above, to start with its the backend integration. The frontend of system A is I should say not part of the exercise or out of scope.
- There is no standard requirement doc. of the same (alas) but the implicit understanding is that the system A should keep on working with existing (A’s only) backend system along with the new system B backend.
So rather in two lines, problem statement can be written as:
Integrate the System A with the backend (input system) of strategic System B, without changing the domain model of it. The project goals are both input should work for System A with minimum changes in it.
Solution
With the above problem statement at hand, to me at onset it looks like a typical Enterprise Integration Problem. Though nobody mentioned that word. And I would discount me any criticism and would say I am not blinded with jumping-the-gun sort of solution architecting. I would go into why I think so now. But do share your views, that’s the reason of this piece.
Software Engineering is about trade-off.
No matter how much I hate the above, I have often come to at peace with above age old adage. So, first I can describe the alternatives and pros and cons of the same.
- The code of System A I have full access to here, so going for Enterprise Integration Patterns (EIP) because of this reason is not valid here. You can say that if I have like blackbox input / output validations or systems tests and/or if the system would have been small I had the only EIP option left, but its not the case.
- However, the major reason for (I can be wrong) me is that it looks to be more of Data Transformation and Input source problem (which is earlier x and now x+y) which doesn’t fit nicely into System A or System B. Hence outside the scope and in the Enterprise Integration Domain. I am also hoping though that I would be able to achieve it with available set of Message Channel, Message Routing, and Message Transformations. If this doesn’t turn out to be that way, it’s bound to be a fail.
- So, the trade-off here is the nice plug-and-play (if I can say) where I can have my System A (and B) working and System A getting new input in the interim and nice movement to the System B backend in the long run. Keeping to the technical benefits, its kind of modular approach. Let’s call it light-weight integration approach only :)
A side note on the Message Sharing way of integration. Other ways to integrate systems / applications would be File Sharing, Database Sharing. Again trade-offs and more trade-offs here. I would argue (even if I am not explaining enough, that could be subject of different post) the message sharing sounds to be least disruptive out in this case.
Misconceptions and Camel Ride
As a bonus I would like to share a misunderstanding I had or rather an experience that you don’t understand somethings till you write it. I hope you would have got by now I have read about EIP and not done it yet in life. So, I used to think that Queues are heavy systems, you can’t have two queues in overall tech stack. So how can you have now. System A had Queue A and system B had Queue B, how can I think of EIP / Camel as a solution. And so now you would have guessed it, this is the story of origin of the post title “two queues and a camel.”
In EIP you can have n number of Queues or Queueing System.
Since we are at it (quotes), one more.
EIP is to Camel what queue is to Kafka.
Camel 101

Now let me move away from the title. That (title) was entirely my problem, my misunderstanding. The next in the series I would like to be some Camel blog series. So, here is the Camel 101.
Camel is EIP (Routing) + Endpoint, Protocol, and Data (Component + Message) abstraction
I have kept EIP first and Component and Message second. Not to demean Components or Message. But that will do for now. At this point I value that more. Also to re-inforce, you can have n Queues, how does it matter if you have not got the EIP and Routing right.
A note about Message before going into Routes and EIP. Message is what decouples system. So we are integrating, but never loose a sight that they (systems) are decoupled to start with and end with. And it’s a good thing. People and systems should talk only that much which is required. In the absence of Message abstraction that would not have been possible. Hence, full marks to Camel. Message is self-describing, complete concept in Camel (and EIP). Starting point, I should say.
Why EIP?
- You can use some patterns, using patterns is good. I would see how far my mileage goes.
- Power of chaining in Routes or composition of Processors. Would also see the same how far I that goes too.
Component
Component is other part of puzzle. So, it is what is at the two end of the Message. The producer / consumer. Or the common abstraction out there. I am hoping that it would be non obtrusive to the Application / Business classes. Still to go more into that (and Camel) in Camel 102.
