Consumer Offsets

Explain Consumer Offsets in Kafka.

The consumer has three options to read messages from the Topic:

  1. --from-beginning — reading all the messages from the beginning.
  2. --latest — will read only those messages that will come after the consumer spun up.

3. --offset — will read from a particular position and particular offset. This option can be done only programmatically.

e.g --offset 6 reads the message in a topic by passing a specific value from the consumer.

  • The first two options --from-beginning and --latest can be explored using the consumer itself.

Let’s Deep dive into Consumer Offset.

  • We have only 1 partition having some records in it. Let’s say we have a Consumer which is going to read the message from the beginning.

For any Kafka consumer, it is required the consumer provide the group id. Now the consumer in general pulls and retrieves multiple records at the same time.

→ As it processes each message it moves the consumer read offset one by one starting from 0,1,2,3…etc.

  • Let say for some reason the consumer is crashed or shut down. While the consumer was down, the producer of the Topic produces some messages.

Question:- Again now the consumer is up after some time. So now how it knows that it needs to read from offset 4?

Reading from offset-4


  • The consumer offset in general are stored in the internal topic __consumer_offsets.
  • Consumer offset behaves like a bookmark for the consumer to start reading the messages from the point it left off.

Practical- Do we really have internal topics?

Please make sure that Zookeeper and Broker instances are running.

→ To check internal topic or consumer offsets we have a command, which is going to list all brokers or list of clusters available in your machine.

.\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --list

__consumer_offsets is the topic that was auto-created
  • __consumer_offsets is the topic which was auto-created for us, which takes care of maintaining the consumer offsets for us.

— Note: test_topic is the topic which is created manually by us.

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Life is a journey of twists and turns, peaks and valleys, mountains to climb and oceans to explore.

Cloud computing: my first steps

Python 3.10 Match — A New Way to Find Patterns

The Essence of Programming

7 DevOps adoption statistics for 2017 [Infographic]

Wikipedia Data Science: Working with the World’s Largest Encyclopedia

Sliding Window #1: Finding the maximum sum of k consecutive elements in an array

Microsoft Introduces the Azure Well-Architected Framework

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sagar Kudu

Sagar Kudu

I am Full Stack Java Developer @ Tata Strive | Get blogs and tutorials related to the (React | Kafka | DevOps) | Follow me on LinkedIn

More from Medium

Kafka for dummies and with practical failure experiments

Deploy java/spring application with mysql DB on Kubernetes

Kafka, Apache Avro and Confluent Platform with Spring Boot

Externalise Kafka Connect Secrets