Consumer Group
Explain Consumer Group In Kafka.
The consumer group-id is mandatory, it plays a major role when it comes to scalable message consumption.
- To start a consumer
group-idis mandatory. group-idplays a major role when it comes to scalable message consumption.
What is message consumption?
let’s consider we have a topic test-topic and 4 partitions. Now we have a consumer-ready with group 1. we have a single consumer pulling all the partitions in the topic and processing them.
The pull loop is always single-threaded, so in this case, a single thread is going to pull from all the partitions.
Case 1: Let’s say the producer of the topic is going to produce messages at a faster rate, the consumer processing rate. In such a case it will introduce a lag in the consumer and you might end up not producing events in real-time. These are where Consumer Groups come in handy.
Case 2: Now let’s say we spend up to another instance of consumer A but make sure that you are using the same group id. Now the partitions are split between the two instances of the consumer partition.
Partition 0 and Partition 1 care is taken by the first instance and Partition 2 and Partition 3 care is taken by the second instance.
- Basically, it means we have scaled a consumption. This will help to process records a little faster than it was before.
Case 3: Let’s make it more by adding two more instances. So we have 4 instances on the Consumer A application but the group id is the same across all the different instances that we have.
- At any given point you are going to process 4 records parallelly. So the consumer groups are the fundamental basis of scalable message consumption.
Case 4: Now let’s say we have 5 consumer instances but only 4 partitions are available for a given topic, in that case, one of the instances will be idle.
As I have mentioned the consumer pool is Single Threaded, in these cases, it will lead to inefficient use of resources.
Real-World Scenario
Let’s take a real-world scenario. We have 4 instances of consumer A with group id 1 and 2 instances of consumer B with group id 2. But the team has to make sure they are not using the same group id. That’s the reason you see a unique group id between the application.
- So the fact is that it is up to the team to decide on how many instances they want.
Summary
- Consumer groups are used for scalable message consumption.
- Each different application will have a unique consumer group.
Who manages the consumer group?
- Kafka manager manages the consumer group.
- Basically, Kafka Broker also acts as a Group coordinator.
Practical Session.
Make sure you are in c:/kafka/bin/windows or simply add .\bin\windows\
How to view consumer groups
- List all the consumer groups available in your local machine.
.\bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092 --list
It gives the result group id is 42743
This is group id — 42743 which gets automatically generated whenever you run a console consumer instance in your machine.
Confused, right? That how id is generated on running consumer instance. Let’s take an example and try to run our previous console consumer command.
e.g Try to run: .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning
output for running consumer console, please note this generates another group id.
After running a consumer console, a new group id is generated for you behind the scene.
one new group id is created.
Instantiate a Kafka consumer with the Consumer Group.
Let’s say you are building a Kafka consumer through code and you want to interact with the Kafka topic. In these cases, it is mandatory for you to provide the group id.
- The above is the same but we have
--group <group-name>
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group <group-name>Step 1: Let’s pick up the consumer id from above.
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group console-consumer-91346→ Run above command now in another cmd.
Running: Instance 1
Now we have a Kafka console consumer instance that is running with a group id 91346
Step 2: Again open a new console and run the above code again.
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group console-consumer-91346
Running: Instance 2
- Now we have two instances of the Kafka consumer with the same group id running and the expectation is that this will have “2 partitions for instance 1 and 2 partitions for instance 2 because we are using the same
test-topicwhen we created a topic we gave 4 partitions.”
Step 3: Now I am going to instantiate a console producer. (new cmd)
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topicStart producing the messages, say 1,2,3,4.
output 1: Message 4 is present in instance 1
output 2: Message 1,2,3 is present in instance 2.
- Basically, they are having the partition split between these two instances of the console consumer.
