Consumer patterns
We write a lot of consumer scripts for queue system, but no one cares how it should be written. People know about MVC, MVP, MVVM approach for writing web application, GUI application. There are a lot of frameworks developed on above philosophy. But there are no guidelines for writing consumers.
We always want to have our business logic decoupled from our infra. Meaning, irrespective of whatever infrastructure we are using to achieve our requirements should not pollute our business logic. Right? So why don’t we try this for writing consumer scripts.
So we will try to write a better consumer using the practises we already know. We will be using java as a mode of communication, but we can communicate over other languages too.
Pre-requisites:
- Basic understanding of any queue system like Rabbitmq, Kafka, SQS etc.
- Zeal for writing clean and decoupled code
- Basic Design patterns like Factory, Repository, Singleton Pattern …
Let’s develop a normal DB updater script for a particular event.
First, we want to decouple our queue provider from our code base, for achieving this we will use Factory pattern which will be responsible for providing in-house queue object instead of Rabbitmq / kafka or other queue object.
Then we will use this Queue for returning from QueueFactory:
Finally we have in house queue system, which is not bound by any provider.
So there is a simple approach for writing modular code, it should be executable from command line, it should not have any infra dependency. Simply meaning, this code can be invoked for http request or queue event. As we first validate our http request in controller then pass the command to our service layer or model layer for fulfilling the business requirements. Same analogy can be used for queue consumers; we can capture the queue event and then, we can delegate this command to our service layer, which will fulfil the business requirements. So, our subscriber code will look like this:
We can use functional interface for passing the callback.
Now we will write our service layer where we can implement our business logic. And also we will inject infra level dependencies from this layer itself like db etc. And we have skipped the db factory implementation as it can be done easily.
Service layer will now handle the interaction with business logic. We can now update the db.
Interacting with db from our consumer script is very normal, but what people do, they write db connector, sql query and other stuff in a single file or script, without thinking about the coupling and cohesion of the code. Think in this way, in future you want to dump the product details to nosql db like mongodb, cassandra etc. Then you will start changing the code for adding support for other dbs, so in a way you are effecting your business logic. For tackling this type of problem, we can leverage on the concept of interfaces between infra layer & business logic layer. On this philosophy there is an architecture called Hexagonal (alternatively Port and Adaptors) which explains the above ideology in detail.
That is why we inject the db reference in above code from subscriber which can be changed from mysql to mongo. And as we are using queue factory, in the same way we are using db factory for decoupling the db dependencies on the service layer.
In service layer we have asked for db and productId as we are about to serialised the product detail from productId, But we have not coupled our db instead we have created ProductRepository and implemented it for MySql.
This way we are asking our repo reference for the data instead the db direct, and we can inject whatever db we want to use.
ProductRepository acts like a contract for Product.
And finally we will implement the ProductRepository for MySql;
By doing the above exercise we have developed a consumer which is liberal for new adoption, and conservative for what it is made.
For better understanding of the above components we can refer this diagram also:
Conclusion:
I hope I explained everything clearly enough for you to understand. If you have any questions, feel free to ask. You can find me on twitter.
Make sure you click on green heart below and follow me for more stories about technology :)
