Kafka SSL : Setup with self signed certificate — Part 1

Understanding and setting up Kafka security is a complex process, I stumbled upon it numerous times and hence thought of creating step by step guide with fine-grained steps so that others don’t have to make the same mistakes. I am not Kafka or SSL expert but whatever knowledge I’ve accumulated so far, this blog post is a reflection of the same.

Terminologies and Descriptions:

1. Certificate Authority (CA)

A certificate authority (CA), is a company or organization that acts to validate the identities of entities (such as websites, email addresses, companies, or individual persons) and bind them to cryptographic keys through the issuance of electronic documents known as digital certificates.

2. Certificate signing request (CSR)

CSR is an encoded message submitted by an applicant to a CA to get an SSL certificate. A certificate authority will use a CSR to create your SSL certificate, but it does not need your private key.

3. KeyStore

KeyStore is used to store private key and identity certificates that a specific program should present to both parties (server or client) for verification.

4. TrustStore

TrustStore is used to determine which certificates (broker or logical client identities) to trust (authenticate).

Following are two examples for TrustStore:

1. The TrustStore contains one or many certificates: The broker or logical client will trust any certificate listed in the TrustStore.
2. The TrustStore contains a Certificate Authority: The broker or logical client will trust any certificate that was signed by the CA in the TrustStore.

Setup Kafka broker:

1. Create own private Certificate Authority (CA)

openssl req -new -newkey rsa:4096 -days 365 -x509 -subj “/CN=Demo-Kafka” -keyout ca-key -out ca-cert -nodes

2. Create Kafka Server Certificate and store in KeyStore:

keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass <password> -keypass <password> -dname “CN=<hostname>” -storetype pkcs12
# verify certificate
keytool -list -v -keystore kafka.server.keystore.jks

3. Create Certificate signed request (CSR):

keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass <password> -keypass <password>

4. Get CSR Signed with the CA:

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-file-signed -days 365 -CAcreateserial -passin pass:<password>
# verify certificate
keytool -printcert -v -file cert-file-signed

5. Import CA certificate in KeyStore:

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass <password> -keypass <password> -noprompt

6. Import Signed CSR In KeyStore:

keytool -keystore kafka.server.keystore.jks -import -file cert-file-signed -storepass <password> -keypass <password> -noprompt

7. Import CA certificate In TrustStore:

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass <password> -keypass <password> -noprompt

8. Setup Kafka broker with KeyStore and TrustStore:

Following is Kafka broker setup using wurstmeister/kafka images and docker-compose, code for the same is available in my public git repository. Please follow README.md for setup cluster and running the sample application. https://github.com/jinternals/kafka_ssl_setup/tree/master/Part%201

version: '3.5'

services:

  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka:2.12-2.2.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: 'SSL://kafka:9092'
      KAFKA_LISTENERS: 'SSL://0.0.0.0:9092'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_SSL_KEYSTORE_LOCATION: '/certs/kafka.server.keystore.jks'
      KAFKA_SSL_KEYSTORE_PASSWORD: 'serverpassword'
      KAFKA_SSL_KEY_PASSWORD: 'serverpassword'
      KAFKA_SSL_TRUSTSTORE_LOCATION: '/certs/kafka.server.truststore.jks'
      KAFKA_SSL_TRUSTSTORE_PASSWORD: 'serverpassword'
      # KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ''
      KAFKA_SECURITY_INTER_BROKER_PROTOCOL: 'SSL'
    volumes:
      - ./server_certs:/certs

Setup Kafka client:

1. Import CA certificate In TrustStore:

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass <password> -keypass <password> -noprompt

2. Setup Kafka client application with TrustStore:

Following is spring boot application config with truststore, code for the same is available in my public git repository. Please follow README.md for setup cluster and running the sample application. https://github.com/jinternals/kafka_ssl_setup/tree/master/Part%201

Multiple Broker SSL Setup

Following should be the topology for your multi-node cluster, every node is the cluster will have their own KeyStore and they will share same TrustStore (CA will be same for all the brokers in a cluster), similarly, clients will share the same TrustStore, Please fork and modify sample application and scripts to setup multi-node cluster (Home Work For You).

Source Code :

https://github.com/jinternals/kafka_ssl_setup/tree/master/Part%201

So far setup we have done is only for encrypting traffic between Kafka broker and client. In the next post, we will see how to authenticate client using SSL.

Next: https://medium.com/@mradulpandey/kafka-ssl-client-authentication-part-2-82c211d64eb6