Set up a Kafka Cluster in Local with multiple Kafka Brokers
In this example, we are going to set up a Kafka Cluster in Local with 3 Kafka Brokers.
we have launched a Kafka broker till now using the following command.
- For Mac or Linux
2. For windows
server.properties is passed as a parameter. This starts up Kafka broker in a machine.
Multiple Brokers in Kafka
→ To start a multiple what needs to be done.
- Create new
server.propertiesfiles for every new broker.
Example: Previous port no. was
9092 and broker-id was
0, Kafka log directory was
Setting up a cluster (configuration)
server.propertiesfiles with the new broker details.
Creating new Broker-1
Follow these steps to add a new broker.
Do the following changes in the file.
- change id to 1
2. Changing port no. to 9093 and auto-create to false
3. change log directory to Kafka-log-1
Creating new Broker-2
Please follow to set up a new broker-2
Starting up these 2 Kafka brokers
Note: Please keep your existing Kafka broker and Zookeeper running.
- starting the first broker
2. starting the second broker
→ So we have successfully started 3 Kafka brokers and now we have a Kafka cluster that is up and running in our machine with 3 brokers.
Creating new Topic
It's time to create a new topic, then we will produce and consume the messages with our new cluster setup.
.\bin\windows\kafka-topics.bat --create --topic test-topic-replicated -zookeeper localhost:2181 --replication-factor 3 --partitions 3
--replication-factor 3is used here and normally it is recommended to use if you are using a cluster setup and this value will be either equal to or less than brokers that you have in a Kafka cluster. Here we have 3 brokers right? So I am going to
- We will topic name to
- The partition we will keep is
partitions 3, we are just keeping in sync with the numbers of brokers that we have. It doesn't matter you can have n number of values you have, I am just giving the partition value 3 here.
Produce the messages using console producer.
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic-replicated
Instantiate a new Consumer to receive the messages.
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic-replicated --from-beginning
Now whatever message we have sent is received to console consumers. Now the interesting part is that we have 3 new Kafka folders right? Let’s go ahead and check that what we have in it.
- close the producer console now and you know have created a
kafka-logs-2directories are created.
- Now each broker got a new folder and that is where it is actually persisting all the messages that are produced to a particular broker. So we have three different directories for each and every broker.
Conclusion: we have successfully a Kafka Cluster with 3 brokers and created a topic in the cluster and successfully produced and consumed the messages into the Kafka cluster.
In the next tutorial, we are going to study How Kafka distributes the client request between the Kafka brokers?
How Kafka Cluster distributes the Client Requests between brokers — LeaderFollower
Before we get into the details of how Kafka distributes a client request, we will discuss how Topics are distributed.