The Content Enricher Pattern is a widely used Enterprise Integration Pattern in implementing a data enrichment workflow. A content enricher takes a data from a structured source message (such as JSON,XML,YAML,Protobuf) and produces an enriched message that is sent further down the data processing pipeline. The content enricher also acts as a gateway to an external resource. Typically the content enricher will create the enriched document using data from an external resource. In this video, I go over how to implement the content enricher messaging pattern in RabbitMQ.
Creating the Content Enrichment Workflow
You can setup the Content Enricher Pattern in RabbitMQ by using two message queues. First message queue is to store the source message and the second message is to store the resulting enriched message. The Content Enricher is an external program that consumes the message from the source message queue. The Content Enricher program retrieves the additional data that will be used to produce the final enriched message. The resulting enriched message will then be published to the result message queue. To create a content enrichment workflow you chain multiple content enrichers together such that the result message queue is the source message queue of another content enricher.
Why RabbitMQ is the right technology for the Content Enricher Pattern
When correctly used, RabbitMQ will guarantee that no messages will be lost. This reliability is the most important property of any message processing workflow. When using RabbitMQ, the Content Enricher program does not need to handle the case when the external resource is not available, such as when a web service you depend on is not reachable. In this scenario, you Content Enricher program do not need to acknowledge message and hence the message will remain in the message queue until that message can be successfully processed.
Communicating with external resources such as calling web services or querying databases are typically slow and using RabbitMQ gives you the advantage of scalability. If your Content Enricher is becoming a bottleneck in this data workflow, you can scale by running multiple Content Enricher instances. It is really that simple when using RabbitMQ!
Don’t forget to check out my Reliable Messaging with RabbitMQ Video Course. It will teach you everything you need to know to implement the Content Enricher Pattern in RabbitMQ.
Originally published at objectzen.com on February 3, 2017.
