Using confluent-kafka on Apple Silicon
Did you try installing confluent-kafka on you new Macbook with the amazing Apple silicon. If the answer is ‘yes’ then you know it’s not a trivial task.
I don’t deny that Apple Silicon is super powerful, but there is still a long way to go. It’s a pain to setup some of the libraries and tools on the new chipset and make them work, well confluent-kakfa is one of them.
What is confluent-kafka
Lets get to the point and see how to set it up !!!
Steps to install confluent-kafka on Apple Silicon devices
- Install Homebrew
Homebrew is a package manager for MacOS. If you don’t have homebrew installed on you mac, copy and paste this command in your terminal to install it.
/bin/bash -c “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
- Install librdkafka
librdkafka is Apache Kafka’s c/c++ library. Use below command to install it. Take a note of the version of the library installed. This will be used in the next step.
brew install librdkafka
- Setup env variables
We need to export some environment variables so that while installing confluent-kakfa, the required c/c++ binaries can be found. Ensure that you change the version of the librdkafka ( 1.8.2 in below command ) based on your machine
Note: Don’t forget to update the librdkafka version in the below export statements.
- Install confluent-kafka
pip install confluent-kafka
These steps when performed correctly will save you from many errors like
- fatal error: ‘librdkafka/rdkafka.h’ file not found
- file not found: linker command failed with exit code 1
Cheers ! Now you can go and focus on actual work rather than spending your time on stackoverflow articles to install confluent-kafka.