ssl

To enable ssl in Kafka, first create CA certificates.

SERVER SETUP

1. Create CA certificate

openssl req -new -newkey rsa:4096 -days 3650 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes

To verify - keytool -printcert -v -file ca-cert

The above command will create 2 certificates ca-cert (public) and ca-key (private key)

2. Set environment variable for password as we will be using password in many places

export SRVPASS=serversecret #any password

3. Generate keys - Have Key store will be created with the below command

keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass $SRVPASS -keypass $SRVPASS -dname "CN=Suresh" -storetype pkcs12

Note: Use public DNS name in the "CN=ec2.**.**" if you are using EC2 instance

Use keytool -list -v -keystore kafka.server.keystore.jks - to list check the key

4. Create a certification request file. This will create a cert-file. Getting certificate is 2 step process. First, is getting the signing request from our key store. This is the file which need to be sent to the Certification authority to get it signed. ( But we are going to sign our own by openssl)

keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass $SRVPASS -keypass $SRVPASS

5. Signing the certificate

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:$SRVPASS

Check the cert - keytool -printcert -v -file cert-signed

6. Create Trust Store - Trust the CA by creating a truststore and importing the ca-cert

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $SRVPASS -keypass $SRVPASS -noprompt

7. Import CA and the signed server certificate into the keystore

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $SRVPASS -keypass $SRVPASS -noprompt

8. Follow the same command for trusted keystore as well ( here we are importing)

keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass $SRVPASS -keypass $SRVPASS -noprompt

9. Verify connection and files - With the above commands we should be having the following files in the directory

Test the connection of the server with the below command

openssl s_client -connect <<hostname or dns>>:9093

Files in the folder

ca-key

ca-cert

cert-file

cert-signed

ca-cert.srl

kafka.server.truststore.jks

kafka.server.keystore.jks

Modify server.proeprties file

Modify listeners address to add the SSL port

Add the ssl section as shown below

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093

advertised.listeners=PLAINTEXT://##your-public-DNS##:9092,SSL://##your-public-DNS##:9093

zookeeper.connect=##your-public-DNS##:2181

ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks

ssl.keystore.password=serversecret

ssl.key.password=serversecret

ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks

ssl.truststore.password=serversecret

CLIENT SETUP

For the clients to connect we can either user the public cert which we created earlier or we can create a new client certification and import the ca-cert. The later would be advisable as we don't require to copy the public key to all the client nodes

1. Set a password (optional) you can pass it directly in the command as well)

export CLIPASS=clientpass

2. Generate the client certificate

Copy the ca-cert which we created earlier to the current directory, then execute the below command

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $CLIPASS -keypass $CLIPASS -noprompt

This will generate kafka.client.truststore.jks file with once certificate imported ( i.e ca-cert)

Verify using

keytool -list -v -keystore kafka.client.truststore.jks

3. Create client.properties and configure SSL parameters

client.properties

security.protocol=SSL

ssl.truststore.location=<path>/kafka.client.truststore.jks

ssl.truststore.password=clientpass

4. Test the producer

~/kafka/bin/kafka-console-producer.sh --broker-list <<your-public-DNS>>:9093 --topic kafka-security-topic --producer.config <path>/client.properties

5. Test the consumer

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server <<your-public-DNS>>:9093 --topic kafka-security-topic --consumer.config <path>/client.properties

Either follow the above steps for creating certificate or this one ( from Kafka website)

CREATE CERTIFICATE

#Step 1

keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey

#Step 2

openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert

keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert

#Step 3

keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:serversecret

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert

keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed