Apache Kafka

Understanding Kafka message format

🔹 Kafka Request–Response Communication Model Kafka clients (Producers, Consumers, Admin) communicate with Kafka Brokers using a binary protocol over TCP. Each client request → Broker processes → sends a corresponding...

Message Size in Kafka

🔹 Default Maximum Message Size in Kafka By default, Kafka has a maximum message size of 1 MB. This is controlled by the broker property: message.max.bytes=1048576 (= 1 MB =...

Kafka Interview Questions

1. How does Kafka ensure exactly-once processing, and what are the challenges involved? Kafka provides exactly-once semantics (EOS) using: Idempotent producers: Ensure retries don’t lead to duplicates. Transactions: Group multiple...

Describe in what ways Kafka enforces security?

Kafka enforces security at four major levels: Authentication, Authorization, Encryption, and Auditing. 🔹 1. Authentication (Who are you?) Kafka verifies the identity of clients (producers/consumers) and brokers using: SSL/TLS Mutual TLS...

What is a Partition in Kafka?

A Partition is the basic unit of storage and parallelism in Kafka. Each Kafka topic is split into one or more partitions. A partition is essentially a log file (append-only)...

What is a Consumer Group in Kafka?

A Consumer Group in Kafka is a collection of one or more consumer instances (processes/threads) that work together to consume messages from one or more topics. Each consumer in the group...