Use cases of Kafka

Kafka is a fault tolerant, scalable, distributed asynchronous messaging queue. It is mostly used when dealing with huge load of data.

Here are the use cases of Kafka:

1. Messaging: Kafka is used for messaging. It works better than the traditional message brokers. Twitter uses it for sending and receiving tweets.
2. Stream processing: Many application using Kafka are using Kafka beyond messaging. Raw data consumed from Kafka is aggregated, enriched and sent to another Kafka topic. For example, Kafka is used for building real time recommendation system.
3. Website activity tracking: At Linkedin, rebuilding user activity tracking pipeline as a real time publish subscribe feeds was the original use case for Kafka. Website activity such as page views, searches are published to Kafka. These feeds are available for subscription by a range of use cases such as real time processing, monitoring and ingesting into Hadoop.
4. Metrics collection: Kafka is often used for operational monitoring data.
5. Log Aggregation: Kafka is used to collect logs from multiple servers. Thus make it available in standard format for multiple consumers.
6. Event sourcing: Event sourcing is a style of application, where state changes are logged in a time-order sequence of records. Kafka would be a best fit as a backend for this kind of applications.
7. Commit log: Kafka can be used as an external commit log for distributed systems. The log helps in replicating the data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data.