Kafka

Sources:https://kafka.apache.org/quickstart

Prerequisites:

    zookeeper

Downloads:

Configuration:

    single node single broker

    single node multiple broker

    multiple node multiple broker

Start:

#Create topics:

kafka-topics.sh --zookeeper <ZOOKEEPER_IP_PORT> --create --topic <TOPIC_NAME> --partitions 1 --replication-factor 1

#Increase nuber of partition

kafka-topics.sh --zookeeper <ZOOKEEPER_IP_PORT> --alter --topic <TOPIC_NAME> --partitions 3

delete topic:

1. Stop Kafka server.

2. Delete the topic directory with rm -rf command.

3. Connect to Zookeeper instance.

4. ls /brokers/topics.

5. Remove the topic folder from ZooKeeper using rmr /brokers/topics/yourtopic.

6. Restart Kafka server.

7. Confirm if it was deleted or not by using this command kafka-topics.sh --list --zookeeper yourip:port.

Consume Messages:

Produce Message:

• using console-producer- tail -n+1 -f <INPUT_FILE> | kafka-console-producer.sh --broker-list <KAFA_SERVER_IP>:<PORT> --topic <TOPIC_NAME> $1 #This will continuously keep producing messages for lines written in and any new line being added to <INPUT_FILE> 

• 
  

Stop:

Problems: 

#1. Following exception thrown while consuming data from a topic where older producer(Nagios) sent test messages, using spring-kafka client-2.0.1 : 

Soln: 

#2. 

Running Kafka on win-10

1. See zookeeper  page, section "Running zookeeper on win-10", for download and zookeeper configuration steps. 

2. Created paths E:\ProgramData\kafka\logs, E:\ProgramData\kafka\config and E:\ProgramData\kafka\data

3. Copied server.properties from installation path to E:\ProgramData\kafka\config\server.properties. and modified - 

log.dirs=E:\\ProgramData\\kafka\\logs

4. Started kafka server in cmd -  E:\>kafka-server-start E:\ProgramData\kafka\config\server.properties

5. Create topic in another cmd -
   C:\Users\rohit>kafka-topics --create --replication-factor 1 --zookeeper localhost:2181 --partitions 2 --topic quizzdata

6. See created topic -
   C:\Users\rohit>kafka-topics --list --zookeeper localhost:2181

7. Run console consumer in cmd -
    C:\Users\rohit>kafka-console-consumer --bootstrap-server localhost:9092 --topic quizzdata

8. Run producer in cmd -
   C:\Users\rohit>kafka-console-producer --broker-list localhost:9092 --topic quizzdata

9. Run consumer to read message from offset (earliest, latest or specific offset) of specific partition -
   C:\Users\rohit>kafka-console-consumer --bootstrap-server localhost:9092 --topic quizzdata --offset earliest --partition 1