Consumer Group

Explain Consumer Group In Kafka.

  • To start a consumer group-id is mandatory.
  • group-id plays a major role when it comes to scalable message consumption.

What is message consumption?

let’s consider we have a topic test-topic and 4 partitions. Now we have a consumer-ready with group 1. we have a single consumer pulling all the partitions in the topic and processing them.

The pull loop is always single-threaded, so in this case, a single thread is going to pull from all the partitions.

Case 1: Let’s say the producer of the topic is going to produce messages at a faster rate, the consumer processing rate. In such a case it will introduce a lag in the consumer and you might end up not producing events in real-time. These are where Consumer Groups come in handy.

Case 2: Now let’s say we spend up to another instance of consumer A but make sure that you are using the same group id. Now the partitions are split between the two instances of the consumer partition.

Partition 0 and Partition 1 care is taken by the first instance and Partition 2 and Partition 3 care is taken by the second instance.

  • Basically, it means we have scaled a consumption. This will help to process records a little faster than it was before.

Case 3: Let’s make it more by adding two more instances. So we have 4 instances on the Consumer A application but the group id is the same across all the different instances that we have.

  • At any given point you are going to process 4 records parallelly. So the consumer groups are the fundamental basis of scalable message consumption.

Case 4: Now let’s say we have 5 consumer instances but only 4 partitions are available for a given topic, in that case, one of the instances will be idle.

As I have mentioned the consumer pool is Single Threaded, in these cases, it will lead to inefficient use of resources.

Real-World Scenario

Let’s take a real-world scenario. We have 4 instances of consumer A with group id 1 and 2 instances of consumer B with group id 2. But the team has to make sure they are not using the same group id. That’s the reason you see a unique group id between the application.

  • So the fact is that it is up to the team to decide on how many instances they want.

Summary

  • Consumer groups are used for scalable message consumption.
  • Each different application will have a unique consumer group.

Who manages the consumer group?

  • Kafka manager manages the consumer group.
  • Basically, Kafka Broker also acts as a Group coordinator.

Practical Session.

Make sure you are in c:/kafka/bin/windows or simply add .\bin\windows\

How to view consumer groups

  1. List all the consumer groups available in your local machine.
.\bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092 --list

It gives the result group id is 42743

This is group id — 42743 which gets automatically generated whenever you run a console consumer instance in your machine.

Confused, right? That how id is generated on running consumer instance. Let’s take an example and try to run our previous console consumer command.

e.g Try to run: .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning

output for running consumer console, please note this generates another group id.

After running a consumer console, a new group id is generated for you behind the scene.

one new group id is created.

Instantiate a Kafka consumer with the Consumer Group.

Let’s say you are building a Kafka consumer through code and you want to interact with the Kafka topic. In these cases, it is mandatory for you to provide the group id.

  • The above is the same but we have --group <group-name>
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group <group-name>

Step 1: Let’s pick up the consumer id from above.

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group console-consumer-91346

→ Run above command now in another cmd.

Running: Instance 1

Now we have a Kafka console consumer instance that is running with a group id 91346

Step 2: Again open a new console and run the above code again.

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --group console-consumer-91346

Running: Instance 2

  • Now we have two instances of the Kafka consumer with the same group id running and the expectation is that this will have “2 partitions for instance 1 and 2 partitions for instance 2 because we are using the same test-topic when we created a topic we gave 4 partitions.”

Step 3: Now I am going to instantiate a console producer. (new cmd)

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic

Start producing the messages, say 1,2,3,4.

output 1: Message 4 is present in instance 1

output 2: Message 1,2,3 is present in instance 2.

  • Basically, they are having the partition split between these two instances of the console consumer.

--

--

--

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn https://www.linkedin.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

My Journey from Back-End to Front-End Development

Introducing the Clearcover API, making car insurance available anywhere

How to choose a static analysis tool

Day 20 — Getting familiar with switch statements in Unity

What makes a technical architect?

(C)ode Libraries: the good, the bad, and the ugly.

What’s the best way to handle the Android camera?

How to build application inside and outside Docker. Dockerfile structure and commands.

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sagar Kudu

Sagar Kudu

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn https://www.linkedin.com

More from Medium

Stream Data From Yugabyte CDC to AWS MSK using Debezium

Kafka DR Strategy Using MirrorMaker 2

Kafka Change Replication Factor to 3

Query Laning in Apache Druid