How do you design an event schema and handle schema evolution over time (Avro/Protobuf + a schema registry)?

#Why this is harder than versioning a REST API

In a request/response API, the client and server are usually deployed close together in time, and a breaking change is at least visible immediately — the next call fails. In an event-driven system, a message published today might sit in a topic (or be replayed from the beginning of one) and get read by a consumer deployed months from now, written by a team that doesn't report to you. The producer and every consumer — present and future — are coupled through the schema, but decoupled in time and deployment. That's the entire problem schema evolution exists to solve.

#The three pieces: format, registry, compatibility rule

Serialization format. Avro? and Protobuf are the two formats you'll see in real systems (JSON is common too, but has no native schema enforcement). Avro schemas are typically declared in JSON and travel with a schema registry ID rather than embedded in every message, which keeps messages compact. Protobuf schemas are declared in .proto files and compiled into language bindings; fields are identified by number, not name, which is the source of most of its evolution rules.

Schema registry. A schema registry (Confluent Schema Registry, AWS Glue Schema Registry, Apicurio) is a versioned store the producer registers a schema against before publishing. The registry assigns a schema ID, the producer embeds only that small ID in the message header/payload, and consumers fetch the corresponding full schema by ID on demand (and cache it). Critically, the registry enforces a compatibility check on registration — a new schema version is rejected outright if it violates the compatibility mode configured for that subject, which stops a breaking change from ever reaching the topic.

Compatibility rule. This is the mode you configure per subject (per topic, essentially):

• BACKWARD — a consumer using the new schema can read data written with the previous schema. This is what lets you upgrade consumers before producers.
• FORWARD — a consumer using the previous schema can read data written with the new schema. This is what lets you upgrade producers before consumers.
• FULL — both hold. This is the safest default for topics with many independent consumer teams, because it makes the producer/consumer upgrade order irrelevant.

#A real Avro schema, v1

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders.v1",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "customerId", "type": "string" },
    { "name": "totalAmountCents", "type": "long" },
    { "name": "currency", "type": "string", "default": "USD" },
    { "name": "placedAt", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    {
      "name": "items",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "OrderItem",
          "fields": [
            { "name": "sku", "type": "string" },
            { "name": "quantity", "type": "int" },
            { "name": "unitPriceCents", "type": "long" }
          ]
        }
      }
    }
  ]
}

#Evolving it: a backward-compatible v2

Six months later, product wants to add a discount and an optional loyalty field, without breaking the dozen consumers already reading this topic:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders.v1",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "customerId", "type": "string" },
    { "name": "totalAmountCents", "type": "long" },
    { "name": "currency", "type": "string", "default": "USD" },
    { "name": "placedAt", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    { "name": "discountCents", "type": "long", "default": 0 },
    { "name": "loyaltyPointsEarned", "type": ["null", "int"], "default": null },
    {
      "name": "items",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "OrderItem",
          "fields": [
            { "name": "sku", "type": "string" },
            { "name": "quantity", "type": "int" },
            { "name": "unitPriceCents", "type": "long" },
            { "name": "discountCents", "type": "long", "default": 0 }
          ]
        }
      }
    }
  ]
}

Both new fields are additive and carry a default, so this change is compatible in both directions: an old consumer reading a v2 message simply never sees discountCents/loyaltyPointsEarned (unknown fields are ignored); a new consumer reading an old v1 message gets the default value for both. That combination — new field, default provided — is what makes a change FULL compatible rather than only one direction.

#What breaks it

#The Protobuf version of the same discipline

Protobuf enforces evolution rules through field numbers rather than names:

Wrap long lines
syntax = "proto3";
package orders.v1;

message OrderPlaced {
  string order_id = 1;
  string customer_id = 2;
  int64 total_amount_cents = 3;
  string currency = 4;
  int64 placed_at_epoch_millis = 5;
  repeated OrderItem items = 6;
  int64 discount_cents = 7;      // added in v2, default 0 is implicit in proto3
  optional int32 loyalty_points_earned = 8; // explicit optional, distinguishes "0" from "absent"

  reserved 9, 10;                 // numbers retired from a field we deleted — never reused
}

message OrderItem {
  string sku = 1;
  int32 quantity = 2;
  int64 unit_price_cents = 3;
}

The rules mirror Avro's: adding a new field number is safe (old binaries ignore unknown field numbers on the wire); never reuse a retired field number for something else, which is why reserved exists — a future engineer adding field 9 for something unrelated would otherwise silently corrupt any message that still has an old binary schema expecting field 9's original meaning. Proto3's implicit zero-value defaults are also a trap: you can't tell "the client sent 0" from "the client didn't set this field" unless you use optional (which adds a presence bit) or a wrapper type.

#Practical evolution discipline

• Treat schemas as an API contract with its own review process and its own repository/CI check — a merged schema change should run compatibility validation against the registry before it can be deployed.
• Default to FULL compatibility mode for any topic more than one team consumes; it removes the need to coordinate producer/consumer deploy order.
• Never delete a field the moment you stop using it — deprecate it (stop populating it, keep it in the schema with its default) for at least one full consumer release cycle before removing it.
• Version the namespace/subject, not necessarily the schema name, when a change truly can't be made compatible (a genuine breaking redesign) — publish OrderPlacedV2 to a new topic or subject and dual-write during migration rather than forcing every consumer to upgrade atomically.