Encoding & Schema Evolution — Data-Intensive Notes

Turning in-memory data structures into bytes — and making sure old and new code can still talk to each other.

Programs work with data in memory as rich structures — objects, lists, hash maps. But to store data on disk or send it over the network, it has to be encoded into a self-contained sequence of bytes. That translation step is where a huge, quiet source of production incidents lives: what happens when the code writing the data and the code reading it disagree about the schema?

This isn’t a hypothetical — it’s the normal state of a system during a rolling deployment, where old and new code run side by side for minutes or hours.

Textual formats

JSON, XML, and CSV are human-readable and ubiquitous, but they’re loose about types: JSON can’t distinguish integers from floats reliably across implementations, and doesn’t support binary strings without a workaround like Base64. They also carry field names as full strings on every single record — wasteful at scale.

Binary formats: Protocol Buffers & Avro

Protocol Buffers and Apache Avro both use a schema to encode data compactly — field names are replaced with small integer tags (Protobuf) or omitted from the payload entirely and resolved via the schema (Avro). This makes messages far smaller and faster to parse, but it makes the schema itself a contract you now have to manage explicitly.

message Order {
  string id = 1;
  int32 customer_id = 2;
  repeated LineItem items = 3;
  // Added later — safe because it's a new tag with a default
  optional string promo_code = 4;
}

The rule that keeps this safe: never reuse a field number, and new fields must be optional (or have a default). Old code encountering promo_code in a newer message it doesn’t understand simply ignores tag 4. New code reading an old message without tag 4 just gets the default. That’s forward and backward compatibility, enforced structurally by the format instead of by discipline alone.

Fig. 4 — Old and new code reading the same wire format

Tag 4 has never been used before. Old code (v1) simply doesn't know about it and skips it — new code (v2) reads it fine. Nothing breaks in either direction during the rollout.

Why this connects back to evolvability

Schema evolution is really the encoding-layer expression of the “maintainability” idea from the first topic: a system is only as easy to change as its data formats let it be. Rigid, ambiguous encoding turns every schema change into a coordinated, risky migration; a well-designed binary format with clear compatibility rules turns it into “add a field, deploy whenever.”