Apache Kafka. MacOS installation guide.

Distributed Systems Development A-Z Guide.

Apache Kafka Logo

Give us a message if you’re interested in Blockchain and FinTech software development or just say Hi at Pharos Production Inc.

Or follow us on Youtube to know more about Software Architecture, Distributed Systems, Blockchain, High-load Systems, Microservices, and Enterprise Design Patterns.

Pharos Production Youtube channel

Preface.

Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

In Kafka, the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backward compatibility with an older version. We provide a Java client for Kafka, but clients are available in many languages.

MacOS Installation Guide.

As usual, it’s really easy to install Kafka on MacOS. Kafka installation consists of three parts. The first thing is to install Java 8. Yes, Java 8. Kafka works only with Java 8. Let’s use Brew to do this. Also if you haven’t installed Cask yet, you should do this now.

brew tap caskroom/cask
brew cask install java8

Next step is to install ZooKeeper — service discovery application. It will be installed together with Kafka. It’s fairly easy to do this.

brew install kafka

Now we’re ready to use Kafka on our local machine. But we will do one more thing — let’s download Kafka archive. We will get configuration files from it. From the link below download the latest version of Kafka. Kafka is created using Scala, so you should choose your favorite Scala version from available Kafka archives. Our choice is Scala 2.12 and Kafka 2.1.0.

Apache Kafka

Kafka download page.

Up & Running.

Let’s start ZooKeeper and Kafka locally. For the configuration of both Kafka and Zoo clusters, we will use config files from the downloaded archive. Untar it simply by double-clicking on it and open the unarchived directory in terminal. Here we have several directories and one of them is a config directory. It contains both Kafka configurations in server.properties and ZooKeeper configurations in zookeeper.properties.

Kafka files and configuration directory.

Before starting ZooKeeper and Kafka broker, let’s change configurations a little. At first, let’s change ZooKeeper data directory. By default, both Kafka and Zoo send all data to /tmp/data. We do not want to mess with /tmp during a development, let’s move all data to the current directory. Create data directory and create zookeeper and kafka directories inside it.

Data directory.

Now in config -> zookeeper.properties change dataDir path for data to newly created zookeeper directory.

ZooKeeper dataDir.

Let’s do the same for Kafka brokers. Open config -> server.properties and change log.dirs to newly created kafka directory. Also, make default number of partitions to 3 — num.partitions — this change will make all new automatically created topics to consist of 3 partitions instead of 1.

Kafka log.dirs

Let’s start ZooKeeper first.

$> zookeeper-server-start config/zookeeper.properties

Everything should go as expected and you see a message in the terminal about 2181 port binding.

ZooKeeper is up and running.

In the second terminal, tab let’s start broker instance — Kafka itself. To do this, simply run from the same directory.

kafka-server-start config/server.properties

Now you should see that the broker is up and running too.

Kafka broker is up and running.

Also, tab with ZooKeeper should show you a new session has been established.

ZooKeeper is watching broker now.

Let’s try it.

To try freshly installed Kafka let’s create a new topic. Run in a separate terminal tab

kafka-topics --zookeeper 127.0.0.1:2181 --topic demo_topic --create --partitions 3 --replication-factor 1

Here we create a new topic — demo_topic — which includes 3 partitions and is not replicated(we have only 1 node for now). Also, command notifies ZooKeeper about a newly created topic, ZooKeeper sits on localhost:2181.

Demo topic created.

In a separate tab let’s create a producer to the current topic

kafka-console-producer --broker-list 127.0.0.1:9092 --topic demo_topic --producer-property acks=all

Here we specify brokers IPs — we have only one listening on localhost:9092, the selected topic and property which will wait for acknowledges from all replicas(only 1 in our case).

Also, let’s create a couple of messages without keys.

Producer created.

Now let’s check our demo_topic and start a consumer.

kafka-topics --zookeeper 127.0.0.1:2181 --topic demo_topic --describe

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic demo_topic -- from-beginning

Immediately we receive two previous messages — this happens because of the flag --from-beginning.

Consumer created.

Now try to send new messages from the producer, the consumer will receive them immediately(immediately means 10ms and more).

To know more about Kafka and how to use Kafka with Java, click on links below

Kafka Using Java. Part 1.

Kafka Using Java. Part 2.

Thanks for reading!