Send and Receive message in Kafka

How to Publish and Subscribe messages (with key) using Kafka?

Learn how you can publish and subscribe to messages in the easiest ways.

Kafka messages sent from the Producers have two properties:

• Key (optional)
• Value

Note: before we have sent message is without keys.

Task

→ We are going to publish all the records, start with the same character should go to the same partition. So that consumers can read the messages in the order they will publish to the topic.

• when the producer is invoked to send the messages, it goes through lots of layers behind the scenes before the message is sent to Kafka. One of the layers is partition.

Using Key: A

In the above diagram, we have used the key- A the key can be of any type, in this example, we are using String.

When a message is sent, Kafka Partioner is going to apply some hashing techniques to determine the partition value and if the same key is sent then it is going to resolve to the same partitions. Partition-0 in these cases.

Now in order to message to the different partitions, we are going to another key say key- B so that it will go to Partition-1and so on…

Note: The same key will resolve messages to the same partitions.

Without Key (earlier method)

kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic

With Key

kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"

Note: The commands for without key and with key are the same, the only difference is that with the key we have two additional properties.

• — property “key.separator=-” e.g key.separator=A
• — property “parse.key=true”

Sending and Receiving messages with Key

• make sure you in c:\kafka the directory (root).

How to instantiate a Console Producer with Key?

Step 1: let’s start our Kafka producer

• open new cmd to send messages

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"

Producer — Creating and sending messages to consumers. (with key — )

→ Now we have sent messages till now, now Kafka consumer is waiting to consume these messages.

How to instantiate a Console Consumer?

Step 2: let’s start our Kafka consumer

• open new cmd to receive messages
• Note that the key sent using"key.separator= - " must be the same in Producer and Consumer. *Here we have used key is -
• Now

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning -property "key.separator= - " --property "print.key=true"

Order is maintained which we sent as earlier.

Note: The rest of the messages is showing null because they are produced as key-value as null and order is not maintained they are totally in the different partitions.

More Kafka Tutorials…

Next →

Explain Consumer Offsets in Kafka.

Previous ←

How to Publish and Subscribe messages using Kafka?