Apache Kafka Tutorial for Beginners: Simple Guide with Real Examples

Apache Kafka is one of the most widely used technologies for building event-driven and distributed applications. If you have ever wondered how multiple services communicate without directly depending on each other, Kafka is a good place to start.

In this tutorial, we will understand Kafka using a simple food delivery example. Then we will move towards topics, partitions, producers, consumers, consumer groups, Spring Boot configuration, Java code, error handling, and real-world use cases.

Author: Deep Singh | Category: Java, Spring Boot, Microservices, Apache Kafka

1. Introduction to Apache Kafka

Imagine that you are building an online food delivery application. A customer places an order, and several things need to happen:

The order service saves the order.
The payment service processes the payment.
The restaurant receives the order.
The notification service sends an SMS or email.
The delivery service starts looking for a delivery partner.

One way to build this application is to make the Order Service call every other service directly. But this creates a dependency between services. If the notification service is temporarily unavailable, the order flow may also be affected if the application handles that failure poorly.

This is where Apache Kafka becomes useful. Instead of directly calling every service, the Order Service can publish an event to Kafka. Other services can independently receive and process that event.

This approach helps us build loosely coupled, event-driven applications. Before writing code, let's understand what Kafka actually is.

2. What Is Apache Kafka?

Apache Kafka is a distributed event streaming platform. It allows applications to publish, store, read, and process streams of events in a reliable and scalable way.

In simple words, Kafka acts like a central event platform. One application can publish an event, and one or more other applications can read that event whenever they are ready.

For example, when a customer places an order, the Order Service may publish:

{
  "orderId": "ORD-101",
  "customerId": "CUST-501",
  "amount": 1499.00,
  "status": "ORDER_PLACED"
}

Kafka stores this event in a topic. The Notification Service, Analytics Service, or another interested application can consume it independently.

Kafka is not simply a database or a traditional message queue. It is designed to handle continuous streams of events, retain records for a configured period, and allow consumers to read those records using offsets.

3. Kafka with a Real-Life Example

Let's continue with our food delivery application. A customer places an order for ₹1,499.

Here is what happens in an event-driven design:

The customer places an order through the application.
The Order Service validates and saves the order.
The Order Service publishes an OrderPlaced event to Kafka.
The Payment Service consumes the event and handles payment processing.
The Notification Service consumes the event and sends a confirmation.
The Analytics Service consumes the same event for reporting.

Notice something important: the Order Service does not need to know how the Notification Service sends an SMS or how the Analytics Service stores reporting data.

It only publishes an event. The interested services decide what to do with that event.

This reduces direct dependencies between services. However, it does not automatically make the complete business process reliable. Each service still needs proper error handling, retries, monitoring, and data consistency.

4. Apache Kafka Architecture

The following diagram shows how different services communicate through Kafka in our food delivery application.

In this architecture, the Order Service acts as a producer. Kafka receives the event and stores it in the orders topic. Different consumer groups can read the same event independently.

For example, the Notification Service can send a confirmation message while the Analytics Service processes the event for reporting.

5. Direct Communication vs Kafka

Let's compare two common approaches to service communication.

Approach 1: Direct REST API Communication

The Order Service calls the Payment Service, Notification Service, and Analytics Service directly using REST APIs.

Order Service
    |
    |---- REST API ----> Payment Service
    |
    |---- REST API ----> Notification Service
    |
    |---- REST API ----> Analytics Service

This approach can work well when an immediate response is required. However, the Order Service becomes responsible for handling the availability and response time of the services it calls.

Approach 2: Communication Through Kafka

Order Service
    |
    |---- Publish OrderPlaced Event ----> Kafka
                                          |
                                          |----> Payment Service
                                          |
                                          |----> Notification Service
                                          |
                                          |----> Analytics Service

With Kafka, the producer and consumers do not need to communicate directly. Consumers can process events independently.

REST API	Apache Kafka
Usually used for request-response communication.	Commonly used for asynchronous event communication.
Caller generally waits for a response.	Producer can publish without waiting for consumer processing.
Services are directly connected through API calls.	Services communicate through events.
Useful when an immediate result is needed.	Useful when events need to be distributed and processed independently.
Retries and persistence need to be designed by the application.	Provides durable event storage according to topic and broker configuration.

Kafka does not replace REST APIs. In real applications, both are often used together. For example, a REST API can accept an order request, while Kafka distributes the resulting business events.

6. Core Kafka Concepts

Before writing Kafka code, you should understand the terms used in almost every Kafka project.

Concept	Meaning	Example
Producer	An application that publishes events to Kafka.	Order Service
Consumer	An application that reads events from Kafka.	Notification Service
Topic	A named stream in which Kafka stores records.	orders
Partition	A topic is divided into partitions for scalability and parallel processing.	orders-0, orders-1
Offset	The position of a record within a partition.	Offset 100
Broker	A Kafka server that stores and serves topic partitions.	Kafka server 1
Consumer Group	A group of consumers that share partition assignments.	notification-group
Replication	Keeping copies of partitions across brokers for fault tolerance.	Replication factor 3
Retention	How long Kafka keeps records or how much data it retains, based on topic configuration.	7 days

7. Producer, Consumer and Consumer Group

What Is a Producer?

A producer is an application that sends records to a Kafka topic. In our example, the Order Service publishes an event whenever an order is placed.

What Is a Consumer?

A consumer is an application that reads records from a Kafka topic. The Notification Service can consume the order event and send a confirmation to the customer.

What Is a Consumer Group?

A consumer group is a group of consumers that work together to consume a topic. Kafka assigns partitions among the active consumers in that group.

For example, suppose the orders topic has three partitions and the Notification Service runs three consumers in the same group. Kafka can assign one partition to each consumer.

A partition is assigned to at most one active consumer within a consumer group at a time. If the group has more consumers than available partitions, some consumers will remain idle.

Now suppose the Payment Service and Notification Service use different consumer groups. Both groups can independently read the same events. Their progress is tracked separately.

8. Topics and Partitions Explained

A topic is divided into partitions. Each partition is an ordered, append-only sequence of records.

Consider an orders topic with three partitions:

Topic: orders

Partition 0:  Order-101, Order-104, Order-107
Partition 1:  Order-102, Order-105, Order-108
Partition 2:  Order-103, Order-106, Order-109

Kafka can distribute records across partitions. This allows multiple consumers in the same group to process different partitions in parallel.

One important point: ordering is guaranteed within a partition, not across all partitions of a topic.

How Does Kafka Select a Partition?

A producer can send a record with a key. Records with the same key are normally mapped to the same partition by the default partitioning strategy, provided the topic's partition count and partitioning setup remain unchanged.

For example, if the customer ID is used as the key, events for the same customer can be kept in the same partition. This is useful when the application needs ordering for that customer.

Partitioning should be designed according to the ordering and workload requirements of the application.

9. How a Kafka Message Flows

Let's understand the complete flow step by step:

The producer creates a record containing a key, value, and optional headers.
The producer selects a topic partition using the configured partitioning strategy.
The record is sent to the Kafka broker that leads that partition.
The broker stores the record according to the topic's configuration.
Depending on the producer's acknowledgment settings, the producer receives confirmation of the write.
A consumer fetches records from its assigned partitions.
The consumer processes the records and manages its offsets.

An offset represents a record's position inside a particular partition. Consumers use offsets to track their reading progress.

Kafka does not delete a record simply because one consumer has read it. Records remain available according to the topic's retention or compaction configuration.

10. Kafka Message Delivery Guarantees

One of the most important topics in Kafka is message delivery semantics. They describe how an application handles message processing and failures.

At-Most-Once Delivery

A record is processed zero or one time. If the consumer commits its offset before processing and then fails, the record may not be processed.

At-Least-Once Delivery

A record is processed one or more times. If processing succeeds but the offset is not committed before a failure, Kafka may deliver the same record again after recovery.

This is a common approach in business applications, but consumers should be designed to handle duplicate events safely.

Exactly-Once Processing

Kafka supports exactly-once processing capabilities for specific transactional workflows, particularly when consuming from Kafka and producing back to Kafka using transactions.

However, this does not automatically guarantee that an external database update, SMS, email, or payment API call happens exactly once. External systems need their own transaction coordination or idempotency strategy.

Producer Acknowledgment Settings

The producer property acks controls how much acknowledgment the producer requires from Kafka for a write.

Setting	Meaning
`acks=0`	The producer does not wait for a broker acknowledgment.
`acks=1`	The leader broker acknowledges the write after accepting it.
`acks=all`	The leader waits for acknowledgment from the required in-sync replicas.

For important business events, acks=all is commonly used along with suitable replication and min.insync.replicas settings. The correct configuration depends on the application's durability and availability requirements.

Remember: producer acknowledgments and delivery semantics are related to reliability, but they are not the same thing.

11. Kafka with Spring Boot

Now let's connect Kafka with Spring Boot. We will create a simple example in which an Order Service publishes an order event and a Notification Service consumes it.

We will use Java 17 and Spring Boot 3.x in this example. Spring Boot's dependency management selects compatible Spring Kafka and Kafka client versions. If you are using a different Spring Boot version, check its dependency compatibility before copying the configuration.

Step 1: Add the Kafka Dependency

For Maven, add the following dependency:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

If you are using Gradle, add:

implementation 'org.springframework.kafka:spring-kafka'

You do not normally need to specify a separate Kafka client version when using Spring Boot's dependency management.

12. Common Spring Boot Kafka Properties

Spring Boot provides convenient configuration properties for Kafka producers, consumers, and listeners. The following are commonly used properties—not an exhaustive list of every Kafka setting.

Property	Purpose
`bootstrap-servers`	Kafka broker addresses used by the application.
`producer.acks`	Controls the acknowledgment requirement for producer writes.
`producer.retries`	Configures producer retry attempts for eligible failures.
`consumer.group-id`	Identifies the consumer group.
`consumer.auto-offset-reset`	Controls where to start when no valid committed offset exists.
`consumer.enable-auto-commit`	Enables or disables automatic offset commits by the Kafka client.
`listener.ack-mode`	Controls the listener container's acknowledgment behavior.
`listener.concurrency`	Configures the number of listener consumer threads.

Example application.yml

spring:
  kafka:
    bootstrap-servers: localhost:9092

    properties:
      security.protocol: PLAINTEXT

    producer:
      acks: all
      retries: 3
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        enable.idempotence: true

    consumer:
      group-id: notification-group
      auto-offset-reset: earliest
      enable-auto-commit: false
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: com.example.kafka.dto
        session.timeout.ms: 10000
        max.poll.interval.ms: 300000

    listener:
      ack-mode: manual
      concurrency: 3
      missing-topics-fatal: false

This is an illustrative local-development configuration. The PLAINTEXT protocol is not suitable for an unsecured production deployment. Configure authentication, encryption, broker replication, access control, and suitable durability settings for your environment.

The spring.json.trusted.packages setting tells the JSON deserializer which Java packages it is allowed to deserialize. Keep this restricted to packages you trust rather than using a wildcard without understanding the security implications.

In this example, the listener uses manual acknowledgment. That means the listener method must receive an acknowledgment object and call acknowledge() after successful processing. We will do that in the consumer example below.

Also remember that auto-offset-reset: earliest is used only when the group has no valid committed offset. It does not force an existing consumer group to read every record again.

13. Project Structure

For learning purposes, we will keep the producer and consumer code in one Spring Boot application. In a real microservices architecture, the Order Service and Notification Service would normally be separate deployable applications.

src/main/java/com/example/kafka
│
├── KafkaApplication.java
│
├── dto
│   └── OrderEvent.java
│
├── producer
│   └── OrderProducer.java
│
├── consumer
│   └── OrderConsumer.java
│
├── controller
│   └── OrderController.java
│
└── config
    └── KafkaTopicConfig.java

14. Create an Order Event

First, we need a Java class that represents the event we want to publish. We will use a Java record to keep the example simple.

Notice that the amount uses BigDecimal instead of double. For financial values, BigDecimal is generally a better choice because floating-point types can introduce precision errors.

package com.example.kafka.dto;

import java.math.BigDecimal;

public record OrderEvent(
        String orderId,
        String customerId,
        BigDecimal amount,
        String status
) {
}

Example event data:

{
  "orderId": "ORD-101",
  "customerId": "CUST-501",
  "amount": 1499.00,
  "status": "ORDER_PLACED"
}

15. Create a Kafka Producer

The producer is responsible for publishing events to a Kafka topic. Spring Kafka provides KafkaTemplate to send messages.

package com.example.kafka.producer;

import com.example.kafka.dto.OrderEvent;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderProducer {

    private static final String TOPIC = "orders";

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderProducer(
            KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publishOrder(OrderEvent orderEvent) {

        kafkaTemplate.send(
                TOPIC,
                orderEvent.orderId(),
                orderEvent
        ).whenComplete((result, exception) -> {

            if (exception != null) {
                System.err.println(
                        "Failed to publish order event: "
                                + exception.getMessage()
                );
                return;
            }

            System.out.println(
                    "Order event published successfully. Offset: "
                            + result.getRecordMetadata().offset()
            );
        });
    }
}

Here, the topic name is orders, and the order ID is used as the message key.

One important detail: KafkaTemplate.send() is asynchronous. It returns a future, so the method does not mean that the broker has already confirmed the message at the moment the call returns.

The completion callback above handles the result or failure of the send operation. In a production application, do not rely only on printing an error. Use structured logging, monitoring, suitable retry policies, and a recovery strategy.

Also, a successful Kafka send does not mean that a consumer has processed the event. Producer acknowledgment and consumer processing are two separate stages.

16. Create a Kafka Consumer

Now let's create a consumer that listens to the orders topic and processes incoming order events.

package com.example.kafka.consumer;

import com.example.kafka.dto.OrderEvent;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Service;

@Service
public class OrderConsumer {

    @KafkaListener(
            topics = "orders",
            groupId = "notification-group"
    )
    public void consumeOrder(
            OrderEvent event,
            Acknowledgment acknowledgment) {

        try {
            System.out.println(
                    "Received order: " + event.orderId()
            );

            System.out.println(
                    "Customer ID: " + event.customerId()
            );

            System.out.println(
                    "Order amount: " + event.amount()
            );

            // Perform the business operation here.
            // For example, send an order confirmation.

            // Acknowledge only after successful processing.
            acknowledgment.acknowledge();

        } catch (Exception exception) {

            // Do not acknowledge a message that failed processing.
            // Let the configured Kafka error handler manage the failure.
            throw exception;
        }
    }
}

The listener receives an event and an Acknowledgment object. After successful processing, it calls acknowledge().

If processing fails, the exception is propagated so that the configured error handler can apply its retry or recovery policy. The exact behavior depends on the listener container and error handler configuration.

This example demonstrates manual acknowledgment. It is not a complete production error-handling setup; we will look at retries and dead-letter topics in the next section.

Example: Publish an Order Using a REST API

To test the producer, we can expose a simple REST endpoint. When the endpoint is called, it creates an order event and publishes it to Kafka.

package com.example.kafka.controller;

import com.example.kafka.dto.OrderEvent;
import com.example.kafka.producer.OrderProducer;
import org.springframework.web.bind.annotation.*;

import java.math.BigDecimal;
import java.util.UUID;

@RestController
@RequestMapping("/orders")
public class OrderController {

    private final OrderProducer orderProducer;

    public OrderController(OrderProducer orderProducer) {
        this.orderProducer = orderProducer;
    }

    @PostMapping
    public String createOrder() {

        OrderEvent event = new OrderEvent(
                "ORD-" + UUID.randomUUID(),
                "CUST-501",
                new BigDecimal("1499.00"),
                "ORDER_PLACED"
        );

        orderProducer.publishOrder(event);

        return "Order event submitted to Kafka";
    }
}

You can call the endpoint using Postman:

POST http://localhost:8080/orders

The producer sends the event to Kafka, and the listener can consume it from the orders topic.

For a real order-creation API, save and validate the order in the database first. Also, do not treat the REST response above as proof that the event was successfully published or that downstream services have completed their work.

17. Error Handling, Retry and Dead Letter Topic

In a real application, message processing can fail for many reasons. For example:

The database may be temporarily unavailable.
An external payment or notification API may time out.
The event may contain invalid data.
A downstream service may be temporarily overloaded.

If a consumer fails to process a message, we need a clear strategy. Simply catching an exception and ignoring it can result in lost business processing.

What Is Retry?

Retry means attempting to process a failed message again. It is useful for temporary failures such as a short database outage or network timeout.

However, retrying every error forever is not a good idea. Invalid messages may never succeed, and repeated retries can block useful work or increase system load.

What Is a Dead Letter Topic?

A Dead Letter Topic (DLT) is a separate Kafka topic where messages can be sent after they fail the configured retry and recovery process. Developers can inspect these messages and decide how to handle them.

orders topic
     |
     v
Order Consumer
     |
     |---- Processing successful ----> Complete
     |
     |---- Processing failed
                 |
                 v
              Retry
                 |
          +------+------+
          |             |
       Success       Retry exhausted
          |             |
          v             v
       Complete     orders.DLT

The diagram shows the general idea. The actual retry count, delay, exception classification, and dead-letter publishing behavior must be configured in the application.

Important Production Considerations

Retry temporary errors, but avoid endlessly retrying permanent errors.
Use a suitable backoff delay between retry attempts.
Send messages that cannot be processed to a dead-letter topic.
Monitor the number of retries and messages in the DLT.
Make consumer operations idempotent so duplicate delivery is safe.
Define how failed messages will be reviewed and replayed.

Spring Kafka provides error handlers and retry-topic support. Choose the approach based on whether you need blocking retries, non-blocking retry topics, or a custom recovery workflow.

18. Why Is the Kafka Message Key Important?

A Kafka record can contain a key and a value. The key is useful when you want related records to be routed to the same partition.

For example, suppose a customer places several orders:

Customer ID: CUST-501

Order-101
Order-102
Order-103

If we use CUST-501 as the message key, Kafka's default partitioning strategy will normally route those records to the same partition, as long as the partitioning setup remains unchanged.

This helps preserve the order of events for that customer within that partition.

In the producer example, we used the order ID as the key:

kafkaTemplate.send(
    "orders",
    orderEvent.orderId(),
    orderEvent
);

Whether you should use order ID, customer ID, account ID, or another value depends on the ordering requirement of your business process.

19. Transactional Outbox Pattern

There is an important problem that often appears when using Kafka with a relational database.

Imagine that the Order Service performs two operations:

Save the order in MySQL.
Publish an OrderPlaced event to Kafka.

What happens if the database save succeeds but the Kafka publish fails? The order exists in the database, but the other services never receive the event.

Now consider the opposite situation: the Kafka event is published, but the database transaction fails. Other services may receive an event for an order that was never successfully saved.

This is commonly called the dual-write problem. A normal database transaction does not automatically make a write to Kafka part of the same atomic transaction.

How Does the Outbox Pattern Help?

The Transactional Outbox Pattern stores the business data and an event record in the same database transaction.

Save the order in the Orders table.
Save an event record in the Outbox table.
Commit both database changes in one transaction.
A separate publisher reads unpublished outbox records.
The publisher sends those events to Kafka.
After successful publishing, the outbox record is marked as published.

Order Request
     |
     v
Order Service
     |
     v
Database Transaction
     |
     +---- Save Order
     |
     +---- Save Outbox Event
     |
     v
Transaction Commit
     |
     v
Outbox Publisher
     |
     v
Apache Kafka
     |
     +---- Payment Service
     |
     +---- Notification Service
     |
     +---- Analytics Service

This approach prevents the order database write and event-record creation from getting out of sync within that database transaction. The publisher can retry sending an outbox event if Kafka is temporarily unavailable.

The publisher may send an event more than once if it crashes after publishing but before marking the record as published. Therefore, consumers should still be idempotent, typically using a unique event ID to detect duplicate processing.

For systems where reliable event publication is important, the Transactional Outbox Pattern is worth considering.

20. Kafka vs RabbitMQ

Kafka and RabbitMQ are both used for asynchronous communication, but they have different design goals and strengths.

Apache Kafka	RabbitMQ
Distributed event streaming platform.	Message broker.
Records are retained according to topic configuration.	Messages are generally removed from a queue after acknowledgment, subject to queue and message settings.
Consumers track their progress using offsets.	Consumers typically acknowledge messages to the broker.
Suitable for event streams, replay, and high-volume data pipelines.	Suitable for routing, task queues, and flexible messaging patterns.
Uses topics and partitions.	Uses exchanges, queues, and bindings.
Supports multiple consumer groups reading the same event stream.	Supports multiple messaging patterns through exchanges and queues.

Neither technology is automatically better for every project. The choice depends on the workload, message-routing requirements, retention needs, replay requirements, operational experience, and infrastructure.

21. When Should You Use Kafka?

Kafka can be a good choice when your application needs:

Communication between multiple independent microservices.
High-volume event ingestion and processing.
Event retention and the ability to replay records.
Multiple independent consumers for the same events.
Streaming data for analytics and monitoring systems.
Asynchronous workflows that should not depend on immediate consumer responses.
Scalable event-driven architecture.

Example Use Cases

Payment status events in fintech applications.
Order and shipment updates in e-commerce applications.
User activity tracking and analytics.
Application logs and monitoring pipelines.
Banking transaction event processing.
Real-time data integration between different systems.

22. When Should You Avoid Kafka?

Kafka is powerful, but it also introduces operational complexity. It is not necessary for every application.

Consider a simpler solution when:

Your application has only a small number of simple communication flows.
You need a straightforward synchronous request-response operation.
Your team does not have the infrastructure or operational capacity to manage Kafka.
A simple database-backed job queue is enough for the requirement.
The application does not need event retention, replay, or distributed event processing.

For example, if a service needs to fetch a customer's account balance and return it immediately, a REST API is generally more suitable than publishing a Kafka event and waiting for a response.

Always start with the business requirement. Do not add Kafka only because the application uses microservices.

23. Kafka Best Practices

Understanding the concepts is important, but production systems also need careful design. Here are some practical points to keep in mind.

1. Choose Meaningful Topic Names

Use names that describe the event or business domain clearly, such as orders, payment-events, or customer-notifications.

2. Design Partitions According to Workload

Partition count affects parallelism and ordering. Consider expected throughput, consumer count, key distribution, and future scaling. Increasing a topic's partition count can change key-to-partition mapping, so plan carefully when ordering matters.

3. Use Appropriate Message Keys

Select a key based on the entity whose events need to remain ordered, such as customer ID, account ID, or order ID.

4. Make Consumers Idempotent

A consumer may receive the same event again after a failure or reprocessing operation. Use a unique event ID or business key to prevent duplicate business actions.

5. Configure Replication Properly

Use suitable replication factors and minimum in-sync replica settings based on your durability and availability requirements.

6. Monitor Consumer Lag

Consumer lag tells you how far a consumer group is behind the latest records in a partition. Increasing lag may indicate slow processing, insufficient consumers, or a downstream bottleneck.

7. Handle Errors Properly

Configure retries, backoff, dead-letter topics, alerting, and a process for replaying failed events.

8. Secure Kafka

Use TLS encryption, authentication, authorization, and restricted network access where appropriate. Avoid exposing Kafka brokers publicly without proper protection.

9. Plan Event Schemas

Keep event structures consistent and plan how they will evolve. Schema Registry with formats such as Avro or Protobuf can help manage compatibility as applications change.

10. Avoid Publishing Sensitive Data Unnecessarily

Do not put passwords, authentication tokens, card data, or unnecessary personal information into Kafka events. Apply data minimization, access control, and retention policies.

24. Apache Kafka Interview Questions

Q1. What is Apache Kafka?

Apache Kafka is a distributed event streaming platform used to publish, store, read, and process streams of events.

Q2. What is the difference between a topic and a partition?

A topic is a named stream of records. A partition is a division of that topic that stores an ordered sequence of records.

Q3. What is a Kafka broker?

A broker is a Kafka server responsible for storing and serving topic partitions.

Q4. What is an offset in Kafka?

An offset is the position of a record within a partition. Consumers use offsets to track their reading progress.

Q5. What is a consumer group?

A consumer group is a set of consumers that share partition assignments and work together to consume a topic.

Q6. Can two consumer groups read the same Kafka topic?

Yes. Different consumer groups can independently read the same topic. Each group maintains its own offsets.

Q7. Can two consumers in the same group read the same partition at the same time?

No. A partition is assigned to at most one active consumer in a consumer group at a time.

Q8. What happens if the number of consumers is greater than the number of partitions?

Some consumers remain idle because a partition can be assigned to only one active consumer in that group at a time.

Q9. Does Kafka guarantee message ordering?

Kafka guarantees ordering within a partition. It does not guarantee a total ordering across all partitions of a topic.

Q10. What is the purpose of a message key?

A message key helps determine the partition to which a record is sent. Records with the same key are normally routed to the same partition by the default partitioning strategy.

Q11. What is the difference between `acks=1` and `acks=all`?

With acks=1, the leader acknowledges the write. With acks=all, the leader waits for acknowledgment from the required in-sync replicas.

Q12. What is consumer lag?

Consumer lag is the difference between the latest available records and the consumer group's processing position. It helps identify consumers that are falling behind.

Q13. What is a Dead Letter Topic?

A Dead Letter Topic stores messages that could not be processed successfully after the configured retry and recovery process.

Q14. Does Kafka delete a message after a consumer reads it?

No. Kafka retains records according to the topic's retention or compaction settings. Reading a record does not normally delete it.

Q15. What is the Transactional Outbox Pattern?

It stores business data and an event record in the same database transaction. A separate publisher sends the event to Kafka, helping avoid inconsistencies caused by writing to a database and Kafka independently.

Q16. Does Kafka guarantee that an external API is called exactly once?

No. Kafka's exactly-once capabilities do not automatically cover external systems such as databases, email providers, or payment APIs. Idempotency and suitable transaction coordination are still required.

Q17. What is the difference between Kafka and RabbitMQ?

Kafka is designed for distributed event streaming and retained event logs. RabbitMQ is a message broker with exchanges, queues, and flexible routing patterns. The right choice depends on the application.

Q18. What happens when a Kafka consumer crashes?

Kafka can rebalance the consumer group and assign partitions to remaining or newly started consumers. Processing may repeat from the last committed offset, depending on the failure and commit timing.

25. Frequently Asked Questions

Is Apache Kafka free to use?

Apache Kafka is open-source software. You can run it on your own infrastructure. Managed Kafka services may have their own pricing.

Can we use Kafka with Spring Boot?

Yes. Spring Kafka integrates Kafka with Spring applications and provides features such as KafkaTemplate, @KafkaListener, listener containers, and error handling.

Can multiple applications consume the same Kafka event?

Yes. Multiple consumer groups can independently consume records from the same topic.

Is Kafka a database?

No. Kafka stores event records for a configured retention period or according to log-compaction settings, but it is not a general-purpose relational database.

Can Kafka be used for payment applications?

Yes. Kafka can distribute payment-related events between services. However, payment systems must carefully handle idempotency, duplicate events, security, audit requirements, transaction consistency, and failure recovery.

Does Kafka replace REST APIs?

No. REST APIs are useful for synchronous request-response operations. Kafka is useful for asynchronous event distribution. Many applications use both.

26. Conclusion

Apache Kafka helps applications communicate through events instead of depending entirely on direct service-to-service calls. It provides a distributed platform for publishing, storing, and consuming event streams.

We started with a food delivery example and then explored producers, consumers, topics, partitions, offsets, consumer groups, message delivery guarantees, and Spring Boot integration.

We also looked at practical concerns such as retry handling, dead-letter topics, message keys, idempotent processing, and the Transactional Outbox Pattern. These are important when moving from a simple tutorial to a real microservices application.

My suggestion is to start with a small Spring Boot producer and consumer project. Publish an order event, consume it from another service, and then experiment with multiple partitions and consumer groups. Once you understand that flow, Kafka's role in a distributed system becomes much easier to understand.

Keep practising, build real projects, and focus on understanding why a particular design is needed—not just how to configure it.

Keep Learning with LearnByDTech

If this tutorial helped you understand Apache Kafka, explore more Java, Spring Boot, Microservices, and System Design tutorials on LearnByDTech.

Have a question or a different approach? Share it in the comments. Learning becomes easier when developers discuss real problems and practical solutions.

Learn. Build. Share.

Pages