Kafka Cluster

Set up a Kafka Cluster in Local with multiple Kafka Brokers

In this example, we are going to set up a Kafka Cluster in Local with 3 Kafka Brokers.

we have launched a Kafka broker till now using the following command.

./kafka-server-start.sh ..config/server.properties

2. For windows

./kafka-server-start.bat ..config/server.properties

Here server.properties is passed as a parameter. This starts up Kafka broker in a machine.

Multiple Brokers in Kafka

→ To start a multiple what needs to be done.

Example: Previous port no. was 9092 and broker-id was 0, Kafka log directory was kafka-logs

Setting up a cluster (configuration)

Example: server.properties

broker.id=1

listeners=PLAINTEXT://localhost:9093

log.dirs=c:/kafka/kafka-logs-1

auto.create.topics.enable=false (optional)

Creating new Broker-1

Follow these steps to add a new broker.

Do the following changes in the file.

2. Changing port no. to 9093 and auto-create to false

3. change log directory to Kafka-log-1

Creating new Broker-2

Please follow to set up a new broker-2

Edit: server-2.properties

broker.id=2

listeners=PLAINTEXT://localhost:9094

log.dirs=c:/kafka/kafka-logs-2

auto.create.topics.enable=false

Starting up these 2 Kafka brokers

Note: Please keep your existing Kafka broker and Zookeeper running.

.\bin\windows\kafka-server-start.bat .\config\server-1.properties

2. starting the second broker

.\bin\windows\kafka-server-start.bat .\config\server-2.properties

Kafka Cluster

→ So we have successfully started 3 Kafka brokers and now we have a Kafka cluster that is up and running in our machine with 3 brokers.

Running 3 brokers simultaneously.

Creating new Topic

It's time to create a new topic, then we will produce and consume the messages with our new cluster setup.

.\bin\windows\kafka-topics.bat --create --topic test-topic-replicated -zookeeper localhost:2181 --replication-factor 3 --partitions 3
a new topic is created

Produce the messages using console producer.

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic-replicated
message sent: Hi

Instantiate a new Consumer to receive the messages.

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic-replicated --from-beginning
message received: Hi

Now whatever message we have sent is received to console consumers. Now the interesting part is that we have 3 new Kafka folders right? Let’s go ahead and check that what we have in it.

Log directories

Conclusion: we have successfully a Kafka Cluster with 3 brokers and created a topic in the cluster and successfully produced and consumed the messages into the Kafka cluster.

In the next tutorial, we are going to study How Kafka distributes the client request between the Kafka brokers?

Software Engineer at HCL | Technical Content Writing | Follow me on LinkedIn https://www.linkedin.com/in/sagarkudu/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store