Send and Receive message in Kafka

How to Publish and Subscribe messages (with key) using Kafka?

Learn how you can publish and subscribe to messages in the easiest ways.

Kafka messages sent from the Producers have two properties:

Note: before we have sent message is without keys.

Task

→ We are going to publish all the records, start with the same character should go to the same partition. So that consumers can read the messages in the order they will publish to the topic.

Using Key: A

In the above diagram, we have used the key- A the key can be of any type, in this example, we are using String.

When a message is sent, Kafka Partioner is going to apply some hashing techniques to determine the partition value and if the same key is sent then it is going to resolve to the same partitions. Partition-0 in these cases.

Now in order to message to the different partitions, we are going to another key say key- B so that it will go to Partition-1and so on…

Note: The same key will resolve messages to the same partitions.

Without Key (earlier method)

kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic

With Key

kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"

Note: The commands for without key and with key are the same, the only difference is that with the key we have two additional properties.

Sending and Receiving messages with Key

How to instantiate a Console Producer with Key?

Step 1: let’s start our Kafka producer

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"
Producer — Creating and sending messages to consumers. (with key — )

Now we have sent messages till now, now Kafka consumer is waiting to consume these messages.

How to instantiate a Console Consumer?

Step 2: let’s start our Kafka consumer

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning -property "key.separator= - " --property "print.key=true"
Order is maintained which we sent as earlier.

Note: The rest of the messages is showing null because they are produced as key-value as null and order is not maintained they are totally in the different partitions.

More Kafka Tutorials…

Next →

Previous ←

Software Engineer at HCL | Technical Content Writing | Follow me on LinkedIn https://www.linkedin.com/in/sagarkudu/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store