Important metrics to Monitor
1. Number of active controller - should always be 1
2. Number of Under replicated Partitions - should always to 0
3. Number of Offile Partitions - should always be 0
Grafana Dashboard - 721 (basic kafka dashboard for kafka)
Open source grafana dashboard from conflentinc
https://github.com/confluentinc/cp-helm-charts/tree/master/grafana-dashboard
Go to the grafana formula and remove the (cp) and the release at the end of formula.
Updating Kafka or restarting kafka
Method1: OLD
When you a are doing kafka we need to do one a at time, bring it down, update and make sure it is up and then start with the next one.
This has to be done one at a time sequentially.
Instead of doing it manually we can use a tool called 'Jolakia' (https://jolokia.org) which is and http agent just like jmx_exporter.
We need to have to find the under replicated partition. curl to 8778 to get the metics.
Method2: New
Since kafka1.1 it is possible to change the configuration without restarting brokers.
Configs are stored in Zookeeper and brokers pick up these configs at runtime or start time
Configs stored in ZK have higher priority than those in server.properties
Following order will be followed for priority of config
1. Dynamic per-broker config stored in ZK
2. Dynamic cluster-wide default config stored in ZK
3. Static broker config from server.properties
4. Kafka default
Kafka.apache.org/documentation/#brokerconfigs
User kafka-configs.sh scripts to update the broker configurations. Some can be updated dynamically some require restart. The table of config shown in the link above will provide the details.