Open in app

Sign in

Write

Sign in

Kafka Encryption with SSL

Ahmet Kaftan
cloudnesil
Published in
4 min readMay 9, 2020

Apache Kafka is a community distributed event streaming platform capable of handling events. This Article is about how you can secure Kafka communication both inter-broker and external. You can find plenty of information in official Confluent web site but through this article you will find useful commands how can you deploy Kafka with SSL encryption. Last but not least I will tell you how to deploy Kafka on top Kubernetes with helm chart of Kafka from canonical incubator helm chart repository.

There are plenty of information and explanation on Confluent web site I will just skip them and get going:

  1. Create Root CA
PASSWORD=cloudnesil
VALIDITY=365
CA_CONFIG=$(echo -n '[dn]\n
CN=CloudNesil\n
[req]\n
distinguished_name=dn\n
[EXT]\n
keyUsage=digitalSignature,keyCertSign\n
basicConstraints=CA:TRUE\n
extendedKeyUsage=serverAuth'| tr -d '\n')openssl req -new -x509 -keyout ca-key -out root-ca -days $VALIDITY -subj '/CN=CloudNesil' -extensions EXT -config <(printf $CA_CONFIG) -passout pass:$PASSWORD

With the command above, you should have two files created: “root-ca” and “ca-key.

2. Create Kafka Server Keystore

SERVER_CONFIG=$(echo -n '[dn]\n
CN=Kafka-brokers\n
[req]\n
distinguished_name=dn\n
[EXT]\n
keyUsage=digitalSignature,keyCertSign\n
extendedKeyUsage=serverAuth\n
subjectAltName=@alt_names\n
[alt_names]\n
DNS.0=kafka.cloudnesil.com\n
DNS.1=kafka-0\n
DNS.2=kafka-1\n
DNS.3=kafka-2\n
DNS.4=kafka-0.kafka-headless\n
DNS.5=kafka-1.kafka-headless\n
DNS.6=kafka-2.kafka-headless'| tr -d '\n')keytool -keystore server.keystore.jks -alias kubernetes -validity $VALIDITY -genkey -dname "CN=CloudNesil" -storepass $PASSWORDkeytool -keystore server.keystore.jks -alias kubernetes -certreq -file server-cert-request -storepass $PASSWORD

You should have two files created: “server.keystore.jks” and “server-cert-request”.

3. Create Server Certifate and Sign

openssl x509 -req -CA root-ca -CAkey ca-key -in server-cert-request -out server-cert-request-signed -days $VALIDITY -CAcreateserial -passin pass:$PASSWORD -extensions EXT -extfile <(printf $SERVER_CONFIG)

You should have two files created: “server-cert-request-signed” and “root-ca.srl”.

4. Import RootCA and Server Certificate to keystore

keytool -keystore server.keystore.jks -alias RootCA -import -noprompt -file root-ca -storepass $PASSWORD
keytool -keystore server.keystore.jks -alias kubernetes -import -noprompt -file server-cert-request-signed -storepass $PASSWORD

You can verify your server.keystore.jks with:

keytool -list -v -keystore server.keystore.jks -storepass $PASSWORD

You should see something like this:

Keystore type: jks
Keystore provider: SUNYour keystore contains 2 entriesAlias name: rootca
Creation date: May 10, 2020
Entry type: trustedCertEntryOwner: CN=CloudNesil
Issuer: CN=CloudNesil
Serial number: b023dd2e7023518d
Valid from: Sun May 10 00:49:31 EET 2020 until: Mon May 10 00:49:31 EET 2021
Certificate fingerprints:
  MD5:  7B:B0:62:20:42:73:5D:8A:8B:27:35:77:5A:0E:FE:E1
  SHA1: 49:74:55:0D:F8:22:84:A9:84:03:53:A0:C6:7F:9E:19:35:59:07:5E
  SHA256: 8F:11:BB:4F:D6:C9:F6:FC:2E:47:77:CB:AE:07:96:65:0F:2D:1C:45:EB:DA:ED:C6:B2:8E:C5:FE:92:C6:7A:19
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3Extensions:#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]#2: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]#3: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_CertSign
]*******************************************
*******************************************Alias name: kubernetes
Creation date: May 10, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=CloudNesil
Issuer: CN=CloudNesil
Serial number: ed6c79362f53647b
Valid from: Sun May 10 01:16:33 EET 2020 until: Mon May 10 01:16:33 EET 2021
Certificate fingerprints:
  MD5:  3C:F8:6A:11:71:A6:FF:BD:67:1B:21:6A:20:23:E2:6A
  SHA1: 59:3F:42:BA:68:61:DB:F9:EB:A5:57:44:F5:96:92:5D:AF:6F:65:51
  SHA256: 5C:A8:D5:1E:3A:04:9D:80:4F:0A:88:F3:41:50:6C:12:24:2A:50:17:7A:D7:0B:29:86:9A:0F:A7:A5:B9:5F:DB
Signature algorithm name: SHA1withRSA
Subject Public Key Algorithm: 2048-bit DSA key
Version: 3Extensions:#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]#2: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_CertSign
]#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: kafka.cloudnesil.com
  DNSName: kafka-0
  DNSName: kafka-1
  DNSName: kafka-2
  DNSName: kafka-0.kafka-headless
  DNSName: kafka-1.kafka-headless
  DNSName: kafka-2.kafka-headless
]Certificate[2]:
Owner: CN=CloudNesil
Issuer: CN=CloudNesil
Serial number: b023dd2e7023518d
Valid from: Sun May 10 00:49:31 EET 2020 until: Mon May 10 00:49:31 EET 2021
Certificate fingerprints:
  MD5:  7B:B0:62:20:42:73:5D:8A:8B:27:35:77:5A:0E:FE:E1
  SHA1: 49:74:55:0D:F8:22:84:A9:84:03:53:A0:C6:7F:9E:19:35:59:07:5E
  SHA256: 8F:11:BB:4F:D6:C9:F6:FC:2E:47:77:CB:AE:07:96:65:0F:2D:1C:45:EB:DA:ED:C6:B2:8E:C5:FE:92:C6:7A:19
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3Extensions:#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]#2: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]#3: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_CertSign
]

5. Create Server Trust Store and Client Trust Store

keytool -keystore server.truststore.jks -alias RootCA -import -noprompt -file root-ca -storepass $PASSWORD
keytool -keystore client.truststore.jks -alias RootCA -import -noprompt -file root-ca -storepass $PASSWORD

You shoould have a file created: “server.truststore.jks” and “client.truststore.jks”.

You can verify your client.truststore.jks with:

keytool -list -v -keystore client.truststore.jks -storepass $PASSWORD

You should see something like this:

Keystore type: jks
Keystore provider: SUNYour keystore contains 1 entryAlias name: rootca
Creation date: May 10, 2020
Entry type: trustedCertEntryOwner: CN=CloudNesil
Issuer: CN=CloudNesil
Serial number: b023dd2e7023518d
Valid from: Sun May 10 00:49:31 EET 2020 until: Mon May 10 00:49:31 EET 2021
Certificate fingerprints:
  MD5:  7B:B0:62:20:42:73:5D:8A:8B:27:35:77:5A:0E:FE:E1
  SHA1: 49:74:55:0D:F8:22:84:A9:84:03:53:A0:C6:7F:9E:19:35:59:07:5E
  SHA256: 8F:11:BB:4F:D6:C9:F6:FC:2E:47:77:CB:AE:07:96:65:0F:2D:1C:45:EB:DA:ED:C6:B2:8E:C5:FE:92:C6:7A:19
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3Extensions:#1: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]#2: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]#3: ObjectId: 2.5.29.15 Criticality=false
KeyUsage [
  DigitalSignature
  Key_CertSign
]

6. Create kubernetes secret

kubectl create secret generic -n kafka kafka-ssl-store --from-file=server.keystore.jks --from-file=server.truststore.jks

7. Deploy Kafka via helm

helm install  my-kafka --namespace kafka  incubator/kafka  --version 0.21.0 \
--set fullnameOverride=kafka \
--set configurationOverrides."advertised\.listeners"=SSL_EXTERNAL_CONNECTION://kafka.cloudnesil.com:"\$((31090 + \${KAFKA_BROKER_ID}))"\\,SSL_INTERNAL://kafka-"\$((\${KAFKA_BROKER_ID}))".kafka-headless:9093 \
--set configurationOverrides."inter\.broker\.listener\.name"=SSL_INTERNAL \
--set configurationOverrides."listener\.security\.protocol\.map"=SSL_EXTERNAL_CONNECTION:SSL\\,SSL_INTERNAL:SSL\\,PLAINTEXT:PLAINTEXT \
--set configurationOverrides."ssl\.keystore\.location"=/var/private/ssl/server.keystore.jks \
--set configurationOverrides."ssl\.keystore\.password"=cloudnesil \
--set configurationOverrides."ssl\.truststore\.location"=/var/private/ssl/server.truststore.jks \
--set configurationOverrides."ssl\.truststore\.password"=cloudnesil \
--set configurationOverrides."ssl\.key\.password"=cloudnesil \
--set external.enabled=true \
--set external.distinct=true \
--set zookeeper.fullnameOverride=kafka-zookeeper \
--set zookeeper.service.type=NodePort \
--set zookeeper.service.ports.client.nodePort=32022 \
--set secrets[0].name=kafka-ssl-store \
--set secrets[0].keys[0]=server.keystore.jks \
--set secrets[0].keys[1]=server.truststore.jks \
--set secrets[0].mountPath=/var/private/ssl

Now you can connect zookeeper from: kafka.cloudnesil.com:32022 and kafka from kafka.cloudnesil.com:31090 using “client.trust.store”, enjoy..!

Kafka
Ssl
Encryption
Secure

--

--

Ahmet Kaftan
cloudnesil

Written by Ahmet Kaftan

16 Followers
Writer for

ahmet.kaftan@cloudnesil.com https://clutch.co/profile/cloudnesil

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams