Kafka is a messaging queue comprising the Kafka broker, producers, and consumers. Producers publish messages to the broker and consumers consume the messages from the broker. Sometimes in the production system, we face issues where consumer suddenly stops consuming and there is an increase in the consumer lag. Let’s understand how we can find the lag of the consumer group.
- Prerequisite
- List all the consumers present in a Kafka Broker
- Kafka Lag using the command line tool
- References
Prerequisite
- Kafka CLI tools
- Kafka
- Kafka Producer must be present
List all the consumers present in a Kafka Broker
First, you need to find all the consumers attached to the Kafka broker, and then only we can see the lag.
./kafka-consumer-groups.sh --bootstrap-server <ip-address:port> --list
Options Used
- –bootstrap-server
- This is the Kafka broker address. It will be a combination of IP address and port
- Your IP address will be different and the port can be different 192.168.18.226:9093
- –list
- Lists all the consumers on a Kafka topic
Two consumers are present in the Kafka broker. we are interested in the consumer-group-a
Kafka Lag using the command line tool
./kafka-consumer-groups.sh --bootstrap-server <ip-address:port> --describe --group <consumer-group-name>
Options Used
- –bootstrap-server
- As already described in the above section
- –describe
- Describe the consumer group and list offset lag (number of messages not yet processed) related to a given group.
- –group <consumer-group-name>
- You need to provide the name of the consumer group and view the lag in the console output
Output Explanation
- The consumer-group-a is attached to the register_user topic
- This topic has three partitions and you can view the LAG column which depicts the lag in each Kafka topic partition
- The total number of messages present can be seen in the LOG-END-OFFSET column
- The current number of messages that the consumer has already read can be seen in the CURRENT-OFFSET column
Raj Ranjan