ssl
To enable ssl in Kafka, first create CA certificates.
SERVER SETUP
1. Create CA certificate
openssl req -new -newkey rsa:4096 -days 3650 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes
To verify - keytool -printcert -v -file ca-cert
The above command will create 2 certificates ca-cert (public) and ca-key (private key)
2. Set environment variable for password as we will be using password in many places
export SRVPASS=serversecret #any password
3. Generate keys - Have Key store will be created with the below command
keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass $SRVPASS -keypass $SRVPASS -dname "CN=Suresh" -storetype pkcs12
Note: Use public DNS name in the "CN=ec2.**.**" if you are using EC2 instance
Use keytool -list -v -keystore kafka.server.keystore.jks - to list check the key
4. Create a certification request file. This will create a cert-file. Getting certificate is 2 step process. First, is getting the signing request from our key store. This is the file which need to be sent to the Certification authority to get it signed. ( But we are going to sign our own by openssl)
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass $SRVPASS -keypass $SRVPASS
5. Signing the certificate
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:$SRVPASS
Check the cert - keytool -printcert -v -file cert-signed
6. Create Trust Store - Trust the CA by creating a truststore and importing the ca-cert
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass $SRVPASS -keypass $SRVPASS -noprompt
7. Import CA and the signed server certificate into the keystore
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass $SRVPASS -keypass $SRVPASS -noprompt
8. Follow the same command for trusted keystore as well ( here we are importing)
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass $SRVPASS -keypass $SRVPASS -noprompt
9. Verify connection and files - With the above commands we should be having the following files in the directory
Test the connection of the server with the below command
openssl s_client -connect <<hostname or dns>>:9093
Files in the folder
ca-key
ca-cert
cert-file
cert-signed
ca-cert.srl
kafka.server.truststore.jks
kafka.server.keystore.jks
Modify server.proeprties file
Modify listeners address to add the SSL port
Add the ssl section as shown below
listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://##your-public-DNS##:9092,SSL://##your-public-DNS##:9093
zookeeper.connect=##your-public-DNS##:2181
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=serversecret
ssl.key.password=serversecret
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=serversecret
CLIENT SETUP
For the clients to connect we can either user the public cert which we created earlier or we can create a new client certification and import the ca-cert. The later would be advisable as we don't require to copy the public key to all the client nodes
1. Set a password (optional) you can pass it directly in the command as well)
export CLIPASS=clientpass
2. Generate the client certificate
Copy the ca-cert which we created earlier to the current directory, then execute the below command
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass $CLIPASS -keypass $CLIPASS -noprompt
This will generate kafka.client.truststore.jks file with once certificate imported ( i.e ca-cert)
Verify using
keytool -list -v -keystore kafka.client.truststore.jks
3. Create client.properties and configure SSL parameters
client.properties
security.protocol=SSL
ssl.truststore.location=<path>/kafka.client.truststore.jks
ssl.truststore.password=clientpass
4. Test the producer
~/kafka/bin/kafka-console-producer.sh --broker-list <<your-public-DNS>>:9093 --topic kafka-security-topic --producer.config <path>/client.properties
5. Test the consumer
~/kafka/bin/kafka-console-consumer.sh --bootstrap-server <<your-public-DNS>>:9093 --topic kafka-security-topic --consumer.config <path>/client.properties
Either follow the above steps for creating certificate or this one ( from Kafka website)
CREATE CERTIFICATE
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:serversecret
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed