Consumer Offsets

Explain Consumer Offsets in Kafka.

The consumer has three options to read messages from the Topic:

  1. --from-beginning — reading all the messages from the beginning.
  2. --latest — will read only those messages that will come after the consumer spun up.

3. --offset — will read from a particular position and particular offset. This option can be done only programmatically.

e.g --offset 6 reads the message in a topic by passing a specific value from the consumer.

  • The first two options --from-beginning and --latest can be explored using the consumer itself.

Let’s Deep dive into Consumer Offset.

  • We have only 1 partition having some records in it. Let’s say we have a Consumer which is going to read the message from the beginning.

For any Kafka consumer, it is required the consumer provide the group id. Now the consumer in general pulls and retrieves multiple records at the same time.

→ As it processes each message it moves the consumer read offset one by one starting from 0,1,2,3…etc.

  • Let say for some reason the consumer is crashed or shut down. While the consumer was down, the producer of the Topic produces some messages.

Question:- Again now the consumer is up after some time. So now how it knows that it needs to read from offset 4?

Reading from offset-4

Answer:

  • The consumer offset in general are stored in the internal topic __consumer_offsets.
  • Consumer offset behaves like a bookmark for the consumer to start reading the messages from the point it left off.

Practical- Do we really have internal topics?

Please make sure that Zookeeper and Broker instances are running.

→ To check internal topic or consumer offsets we have a command, which is going to list all brokers or list of clusters available in your machine.

.\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --list

__consumer_offsets is the topic that was auto-created
  • __consumer_offsets is the topic which was auto-created for us, which takes care of maintaining the consumer offsets for us.

— Note: test_topic is the topic which is created manually by us.

--

--

--

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn https://www.linkedin.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

The story of Resis

Setting up Nginx, Gunicorn, Celery, Redis, Supervisor, and Postgres with Django to run your Python…

What I realized after 100 leetcode SQL questions…

Website Rollout Feature Updates Zero Downtime

Shut down the computer with a CMD script

MY JOURNEY TO BECOME ANDROID DEVELOPER

Resources on open-everything and sustainable model

Terraform — Best practices and project setup

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sagar Kudu

Sagar Kudu

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn https://www.linkedin.com

More from Medium

Single Source of Truth : Business Process Automation With Camunda

Query Laning in Apache Druid

Redis Cluster: Architecture and Replication

Data Replication (CDC) from Oracle to PostgreSQL using Debezium, Run in Docker & Exposed in Grafana