Kafka Cluster distributes the Client Requests between brokers
How Kafka Cluster distributes the Client Requests between brokers — LeaderFollower?
Before we get into the details of how Kafka distributes a client request, we will discuss how Topics are distributed.
We have a zookeeper and Kafka cluster. In this example, we have a cluster with 3 brokers.
- out of the 3 brokers, one broker will behave as a Controller. Normally this is the first broker to join the cluster. Think of this as an Additional role for the broker. Now we have the environment completely set.
- Now its time to create a
topicand create-topic command is issued to the zookeeper. The zookeeper takes care of redirecting the request to the controller.
→ The role of this Controller is to distribute the ownership of the partitions to the available broker.
→ Now in this example, we have partition-0 sitting in broker-1, partition-1 in broker-1 and partition-2 in broker-2.
- In a distributed system, the concept of distributed partitions to the brokers is called leader assignment. So in a nutshell the topic is distributed across the Kafka cluster.
How cluster handle the connection from the client’s?
→ let’s look at the producer first. The producer has a layer called a Partitioner which takes care of determining which partition the message is going to go.
Message- ABC
So the producer sends the first message ‘ABC’, it goes to the partitioner before the message is sent to the Kafka topic.
The partitioner determines this message should go to partition-0. In this case the leader of partition-0 is broker-1 . So the message will be sent to the broker-1.
The client will always invoke the leader of the partition. After that, the message ‘ABC’ is persisted into the file system of broker-1. This process is repeated for the next following messages, say ‘DEF’.
Message- DEF
Now we have the message ‘DEF’, it goes to the partitioner. The partitioner resolves the partition-1 and the leader of the partition-1 is broker-2.
So the request is directed to broker-2 and it gets persisted into the file system of the broker-2.
Message- GHI
After sending the message “GHI” it again follows the same step. The message result in partition-2 and the leader of the partition-2 is broker-3.
So the message is directed to the broker-3. So you can as client request from the producer and it is distributed between the brokers based on the partition, which indirectly means that the load is distributed between the brokers.
How the client request is distributed from the Kafka consumer end?
we have the consumer ready to pull test-topic-replicated . When the pull loop is executed, the request goes to all the partitions and retrieves the records from them.
Here, each broker owns a partition. In this case, the pull loop goes to all the brokers and retrieves the record from them, and the retrieved record is handed over to the consumer, and the consumer processes this request successfully, and the same flow repeats.
→ So in a nutshell even from the consumer end the request to retrieve the data are distributed between the brokers.
Basically, the client call will only go to the partition leader of the topic and retrieve the data.
Kafka Consumer flow with Kafka Consumer Groups.
As it is common practice to run multiple instances of the consumer and process the records from the Kafka topic.
So here we have 3 instances of the consumer with the same group id. In the concept of the consumer group, if there are one or more instances of the consumer with the same group id, then the partition is distributed for scalable message consumption.
- Here each consumer instance has one partition assigned. When the pull loop gets executed, each instance is going to pull the data from the partition that they are interested in pull call goes to the leader of the partition of the topic and retrieves the data.
- So in here, the partition-0 calls go to the local-1 and partition-1 calls go to broker-2 and so on, and the process repeats.
Summary
- partition leaders are assigned during topic creation.
- Any clients irrespective of producer or consumer will invoke the leader of the partition to produce or consume, with this the load is evenly distributed by invoking the partition leaders.
