• Apache ZooKeeper is a distributed coordination service used by Apache Kafka (in versions before KRaft mode).

• It helps Kafka brokers work together as a cluster by storing metadata and managing cluster state.

Think of ZooKeeper as the “manager” that keeps track of who the leaders are, which brokers are alive, and where partitions live.

🔹 Why Kafka Needs ZooKeeper (Pre KRaft)

Kafka is a distributed system with multiple brokers. To keep everything consistent, Kafka uses ZooKeeper to:

1. Cluster Management

   • Maintains the list of active brokers in the cluster.

   • Helps new brokers register themselves when they start.

2. Metadata Storage

   • Stores information about topics, partitions, replicas, and configuration.

3. Leader Election

   • Each partition has a leader replica (handles reads/writes).

   • ZooKeeper helps elect a new leader if the current one fails.

4. Failure Detection

   • Tracks broker heartbeats.

   • If a broker goes down, ZooKeeper notifies Kafka to trigger failover.

5. Configuration Management

   • Stores topic-level configs (e.g., replication factor, partition count).

   • Brokers fetch configs from ZooKeeper.

🔹 ZooKeeper in Action (Example Flow)

1. Broker joins → registers itself in ZooKeeper.

2. Producer sends message → ZooKeeper helps broker know which partition leader to write to.

3. If leader broker crashes → ZooKeeper coordinates a new leader election from ISR.

4. Consumers ask ZooKeeper indirectly (through Kafka) to discover partition leaders.

🔹 Limitations of ZooKeeper in Kafka

• Extra dependency (separate cluster to maintain).

• Scalability bottleneck for very large Kafka clusters.

• Complexity in leader election and metadata synchronization.

🔹 New Approach: KRaft Mode

• Since Kafka 2.8, Kafka introduced KRaft (Kafka Raft Metadata mode) → eliminates ZooKeeper.

• Now Kafka itself manages metadata and quorum using the Raft consensus algorithm.

• Benefits: simpler ops, better scalability, faster failover.

🔹 In Short

• ZooKeeper = coordination & metadata service for Kafka.

• Used for: cluster management, leader election, metadata storage, failure detection.

• Required in older Kafka versions, but replaced by KRaft in modern Kafka.