What is Kafka Schema Registry?

Kafka Schema Registry is a service that manages and enforces schemas for Kafka messages.

  • A schema defines the structure (fields, data types, defaults) of the message.
  • Without it, producers and consumers might break if the message format changes.
  • The Schema Registry ensures compatibility, validation, and governance of data.

👉 It’s most often used with Avro, but also supports JSON Schema and Protobuf.

🔹 Why use Schema Registry?

  1. Consistency → All producers/consumers agree on message structure.
  2. Evolution → Allows schema changes (adding/removing fields) without breaking apps.
  3. Compact messages → Instead of sending full schema each time, Kafka only sends a schema ID; the consumer fetches the actual schema from the registry.
  4. Governance → Central place to manage versions and compatibility rules.

🔹 How it Works

  1. Producer

    • Uses Avro/JSON/Protobuf to serialize data.

    • Sends message to Kafka topic with a schema ID in the header.

    • Schema itself is stored once in Schema Registry.

  2. Consumer

    • Reads schema ID from message.

    • Fetches the schema from Schema Registry.

    • Deserializes message safely.

🔹 Example: Avro + Schema Registry

1. Define a Schema (user.avsc)

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

2. Producer (Java with Confluent Serializer)

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://localhost:8081");

Producer<String, GenericRecord> producer = new KafkaProducer<>(props);

Schema schema = new Schema.Parser().parse(new File("user.avsc"));
GenericRecord user = new GenericData.Record(schema);
user.put("id", 1);
user.put("name", "Alice");
user.put("email", "alice@example.com");

producer.send(new ProducerRecord<>("users", "key1", user));

👉 Here, only the schema ID is sent in Kafka, not the full schema.

🔹 Schema Evolution & Compatibility

Schema Registry enforces compatibility modes when new versions are registered:

  • BACKWARD → New schema can read old data.

  • FORWARD → Old schema can read new data.

  • FULL → Both forward + backward compatible.

  • NONE → No compatibility check.

Example:

  • Add a new optional field → usually backward compatible.

  • Remove a required field → breaks backward compatibility.

🔹 Quick REST API Calls

Check schemas via Schema Registry API:

  • List subjects:

    curl http://localhost:8081/subjects

  • Get latest schema:

    curl http://localhost:8081/subjects/users-value/versions/latest

🔹 In Short

Kafka Schema Registry = a central service to:

  • Store message schemas (Avro, JSON, Protobuf).

  • Assign schema IDs for compact Kafka messages.

  • Enforce compatibility rules for schema evolution.

👉 Ensures safe, evolvable, strongly-typed event streams across producers and consumers.

Back to blog

Leave a comment