Apache Kafka. MacOS installation guide.
Distributed Systems Development A-Z Guide.
Give us a message if you’re interested in Blockchain and FinTech software development or just say Hi at Pharos Production Inc.
Or follow us on Youtube to know more about Software Architecture, Distributed Systems, Blockchain, High-load Systems, Microservices, and Enterprise Design Patterns.
Preface.
Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
In Kafka, the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backward compatibility with an older version. We provide a Java client for Kafka, but clients are available in many languages.
MacOS Installation Guide.
As usual, it’s really easy to install Kafka on MacOS. Kafka installation consists of three parts. The first thing is to install Java 8. Yes, Java 8. Kafka works only with Java 8. Let’s use Brew to do this. Also if you haven’t installed Cask yet, you should do this now.
brew tap caskroom/cask
brew cask install java8
Next step is to install ZooKeeper — service discovery application. It will be installed together with Kafka. It’s fairly easy to do this.
brew install kafka
Now we’re ready to use Kafka on our local machine. But we will do one more thing — let’s download Kafka archive. We will get configuration files from it. From the link below download the latest version of Kafka. Kafka is created using Scala, so you should choose your favorite Scala version from available Kafka archives. Our choice is Scala 2.12 and Kafka 2.1.0.
Up & Running.
Let’s start ZooKeeper and Kafka locally. For the configuration of both Kafka and Zoo clusters, we will use config files from the downloaded archive. Untar it simply by double-clicking on it and open the unarchived directory in terminal. Here we have several directories and one of them is a config directory. It contains both Kafka configurations in server.properties and ZooKeeper configurations in zookeeper.properties.
Before starting ZooKeeper and Kafka broker, let’s change configurations a little. At first, let’s change ZooKeeper data directory. By default, both Kafka and Zoo send all data to /tmp/data. We do not want to mess with /tmp during a development, let’s move all data to the current directory. Create data directory and create zookeeper and kafka directories inside it.
Now in config -> zookeeper.properties change dataDir path for data to newly created zookeeper directory.
Let’s do the same for Kafka brokers. Open config -> server.properties and change log.dirs to newly created kafka directory. Also, make default number of partitions to 3 — num.partitions — this change will make all new automatically created topics to consist of 3 partitions instead of 1.
Let’s start ZooKeeper first.
$> zookeeper-server-start config/zookeeper.properties
Everything should go as expected and you see a message in the terminal about 2181 port binding.
In the second terminal, tab let’s start broker instance — Kafka itself. To do this, simply run from the same directory.
kafka-server-start config/server.properties
Now you should see that the broker is up and running too.
Also, tab with ZooKeeper should show you a new session has been established.
Let’s try it.
To try freshly installed Kafka let’s create a new topic. Run in a separate terminal tab
kafka-topics --zookeeper 127.0.0.1:2181 --topic demo_topic --create --partitions 3 --replication-factor 1
Here we create a new topic — demo_topic — which includes 3 partitions and is not replicated(we have only 1 node for now). Also, command notifies ZooKeeper about a newly created topic, ZooKeeper sits on localhost:2181.
In a separate tab let’s create a producer to the current topic
kafka-console-producer --broker-list 127.0.0.1:9092 --topic demo_topic --producer-property acks=all
Here we specify brokers IPs — we have only one listening on localhost:9092, the selected topic and property which will wait for acknowledges from all replicas(only 1 in our case).
Also, let’s create a couple of messages without keys.
Now let’s check our demo_topic and start a consumer.
kafka-topics --zookeeper 127.0.0.1:2181 --topic demo_topic --describe
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic demo_topic -- from-beginning
Immediately we receive two previous messages — this happens because of the flag --from-beginning.
Now try to send new messages from the producer, the consumer will receive them immediately(immediately means 10ms and more).
To know more about Kafka and how to use Kafka with Java, click on links below
Thanks for reading!