Send and Receive message in Kafka
How to Publish and Subscribe messages (with key) using Kafka?
Kafka messages sent from the Producers have two properties:
- Key (optional)
Note: before we have sent message is without keys.
→ We are going to publish all the records, start with the same character should go to the same partition. So that consumers can read the messages in the order they will publish to the topic.
- when the producer is invoked to send the messages, it goes through lots of layers behind the scenes before the message is sent to Kafka. One of the layers is partition.
In the above diagram, we have used the
key- A the key can be of any type, in this example, we are using
When a message is sent, Kafka
Partioner is going to apply some hashing techniques to determine the partition value and if the same key is sent then it is going to resolve to the same partitions.
Partition-0 in these cases.
Now in order to message to the different partitions, we are going to another key say
key- B so that it will go to
Partition-1and so on…
Without Key (earlier method)
kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic
kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"
Note: The commands for without key and with key are the same, the only difference is that with the key we have two additional properties.
- — property “key.separator=-” e.g key.separator=A
- — property “parse.key=true”
Sending and Receiving messages with Key
- make sure you in
c:\kafkathe directory (root).
How to instantiate a Console Producer with Key?
Step 1: let’s start our Kafka producer
- open new cmd to send messages
.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"
→ Now we have sent messages till now, now Kafka consumer is waiting to consume these messages.
How to instantiate a Console Consumer?
Step 2: let’s start our Kafka consumer
- open new cmd to receive messages
- Note that the key sent using
"key.separator= - "must be the same in Producer and Consumer. *Here we have used key is
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-topic --from-beginning -property "key.separator= - " --property "print.key=true"
Note: The rest of the messages is showing
null because they are produced as key-value as null and order is not maintained they are totally in the different partitions.