Messaging Systems : ETL for real time Data Processing

Problem

With Data revolution happening very rapidly and terabytes of structure and unstructured data being generated every hour in form logs, user data, audio and video and etc. Batched process and ETL(Extract, Transform and Load) are not able meet the real time data processing goals of the organizations.

Solution

Messaging systems

The first era of messaging systems brought us the point to point to messaging systems which has one source and one destination for each message.

point-to-point messaging system. courtesy oracle docs

As the data is growing with invent of IOT multiple senders and multiple receivers for similar kind of messages it became impossible to have develop, maintain a point-to-point messaging system which let to invent of a publish and subscribe (pub/sub) messaging system.

Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication. In a pub/sub model, producer (sender) publishes a message to a topic is immediately received by all of the subscribers(receiver) to the topic. Pub/sub messaging can be used to enable event-driven architectures, or to decouple applications in order to increase performance, reliability and scalability.