Encoding and Evolution

Backward and Forward Compatibility

Code changes can't happen instantaneously:

• Server-side: you do a rolling upgrade (staged rollout), deploy to a few nodes at a time, monitor, continue. No downtime, encourages frequent releases.
• Client-side: users may not install updates for a long time.

So old and new code (and old and new data) coexist. You need:

Fullscreen

For APIs: an older client calling a newer service needs backward compatibility on the request + forward compatibility on the response. A newer client calling an older service needs the reverse.

The forward-compatibility data-loss trap (Figure 5-1):

Fullscreen

If the old code decodes into a model object that doesn't preserve unknown fields, the new field is silently dropped on rewrite. Good formats let old code pass through fields it doesn't understand.

---

Formats for Encoding Data

Programs hold data two ways: in memory (objects, structs, lists, pointers, optimized for the CPU) and as a byte sequence (for files/network, self-contained, no pointers).

Fullscreen

• Encoding = in-memory → bytes (a.k.a. serialization, marshaling). Decoding = the reverse (parsing, deserialization, unmarshaling). The book uses "encoding" to avoid clashing with transaction serializability (Ch. 8).
• Zero-copy formats (Cap'n Proto, FlatBuffers) avoid the conversion step entirely, usable directly at runtime and on disk/network.

Language-specific formats

Java Serializable, Python pickle, Ruby Marshal, Kryo. Convenient (restore objects with little code) but have deep problems:

• Language lock-in, hard to read in another language; precludes cross-organization integration.
• Security, decoding can instantiate arbitrary classes → remote code execution if an attacker controls the bytes.
• Versioning is an afterthought, poor forward/backward compatibility.
• Efficiency is poor (Java's built-in serialization is notoriously slow and bloated).

→ Only use for very transient purposes.

JSON, XML, CSV (textual)

Widely supported and human-readable, but flawed:

• XML is verbose and over-complicated; CSV has no nesting and vague escaping rules.
• Number ambiguity: XML/CSV can't tell a number from a digit-string; JSON can't distinguish integers from floats or specify precision. Integers > 2^53 break in JavaScript (IEEE 754 doubles), X returns post IDs twice (as a number and a string) to work around this.
• No binary strings, people Base64-encode (works, but +33% size).
• Their schema languages (XML Schema, JSON Schema) are powerful but complex.

Despite flaws, they're fine for many uses, especially data interchange between organizations, agreement matters more than elegance.

JSON Schema

Widely used (OpenAPI, schema registries, PostgreSQL/MongoDB validators). Has primitive types plus a validation spec (e.g., a port between 1 and 65,535).

• Open content model (additionalProperties: true, the default): any undefined field is allowed → schema mostly defines what isn't permitted.
• Closed content model (additionalProperties: false): only explicitly defined fields allowed.
• Powerful (conditional if/else, named types, remote references) but can be unwieldy and hard to evolve compatibly.

Binary variants of JSON/XML

MessagePack, CBOR, BSON, etc. More compact and sometimes faster, but since they keep field names in the data (no schema), the savings are small. Example record encoded in MessagePack = 66 bytes vs 81 bytes textual JSON, modest, debatable whether it's worth losing readability.

Protocol Buffers (and Thrift)

Schema-driven binary format from Google (Thrift is Facebook's analog; mostly the same ideas). You define a schema in an IDL (interface definition language); a code generator produces classes in many languages.

Encoded record = 33 bytes (vs 66 for MessagePack). Key trick: field tags, small numbers (1, 2, 3) replace field names in the encoded data. Each field carries a type annotation + tag packed into a byte, and variable-length integers (e.g., 1337 in two bytes). repeated indicates a list. No field names appear in the bytes.

Schema evolution rules:

• You can rename a field freely (names aren't in the data); you can never change a field's tag.
• Add a field: give it a new tag. Old code ignores unknown tags (the type annotation tells it how many bytes to skip) → forward compatible. New code reading old data fills missing fields with defaults → backward compatible.
• Remove a field: never reuse its tag (old data still references it); reserve retired tags.
• Change datatype: sometimes possible, but values may be truncated (e.g., 64-bit → 32-bit).

Avro

Schema-driven binary format (born 2009 as a Hadoop subproject). Two schema languages: Avro IDL (human) and a JSON-based one (machine). Encoded record = 32 bytes, the most compact here.

The radical difference: no field tags and no field names in the data. It's just concatenated values. You can only decode by knowing the exact schema used to write it.

Fullscreen

• Writer's schema: the version used to encode the data.
• Reader's schema: the version the reading app expects, can differ. Avro performs schema resolution, matching fields by name (order doesn't matter). Fields in the writer's schema but not the reader's are ignored; fields in the reader's but not the writer's get the reader's default value.

Avro evolution rules:

• Forward compat = writer uses a newer schema than reader. Backward compat = writer uses an older schema than reader.
• You may only add or remove a field that has a default value. (Add a no-default field → breaks backward compat; remove a no-default field → breaks forward compat.)
• null is not a default unless you use a union type with null as the first branch (explicit about nullability → prevents bugs).
• Rename a field via reader-side aliases (backward compatible, not forward). Adding a union branch is backward but not forward compatible.

How does the reader know the writer's schema? (You can't embed the full schema per record, it'd dwarf the data.)

• Large file, many records (Avro object container files): include the schema once at the top.
• Database of individual records: store a version number per record + a registry of schema versions (Confluent Schema Registry, LinkedIn Espresso).
• Network connection: negotiate the schema version at connection setup (Avro RPC).

Dynamically generated schemas are Avro's big advantage: because there are no tag numbers, you can auto-generate an Avro schema from a relational schema and re-generate it whenever the DB schema changes, fields match by name, no manual tag assignment (which Protocol Buffers would require).

The merits of (binary) schemas

• More compact than binary-JSON variants (omit field names).
• The schema is always-up-to-date documentation (required for decoding).
• A schema registry lets you check compatibility before deploying.
• Code generation enables compile-time type checking in statically typed languages.
• Net: schema evolution gives schema-on-read flexibility plus better guarantees and tooling. (ASN.1 had these ideas in 1984, still used for X.509 SSL certificates, but is too complex for new use.) Keep the number of concurrent schema formats small.

---

Modes of Dataflow

Compatibility is a relationship between an encoder and a decoder. Who encodes and decodes depends on how data flows.

Fullscreen

Through databases

• The writer encodes; the reader decodes (possibly your future self → backward compatibility needed). Multiple app versions hit the DB at once during rolling upgrades → forward compatibility also needed (preserve unknown fields, Figure 5-1).
• Data outlives code: an app is fully replaced in minutes, but five-year-old data stays in its original encoding unless rewritten. Migrating large data is expensive, so DBs defer it (LSM compaction rewrites in the latest format; relational DBs add a nullable column without rewriting and fill nulls on read). The DB appears to use one schema even though storage mixes historical versions.
• Archival storage: dumps/snapshots are written in one go with the latest schema → great fit for Avro object container files or columnar Parquet.

Through services: REST and RPC

• Client/server: servers expose an API (a "service") over the network; clients call it. Unlike a database (arbitrary queries), a service exposes an application-specific API, encapsulation and fine-grained control over what clients can do.
• Web service = service using HTTP. Used by mobile/browser clients, service-to-service within an org, and cross-organization APIs (payments, OAuth).
• REST = a design philosophy built on HTTP (URLs for resources, HTTP features for caching/auth/content negotiation); an API following it is RESTful.
• Clients need to know endpoints and formats → service IDLs document and evolve APIs: OpenAPI/Swagger (JSON web services) and Protocol Buffers (gRPC). Frameworks (Spring Boot, FastAPI, gRPC) handle routing/metrics/auth so you write business logic; some generate the IDL from code, others generate code from the IDL.

The problems with RPC. RPC (since the 1970s) tries to make a remote call look like a local function call (location transparency), but a network call is fundamentally different:

• Unpredictable, requests/responses can be lost; the remote machine may be slow or down.
• Timeouts, you may get no result and not know whether the request was processed.
• Retries can duplicate the action unless you build idempotence in.
• Latency is wildly variable (sub-ms to seconds).
• Parameters must be encoded as bytes (no passing local pointers); problematic for large/mutable objects.
• Cross-language type mismatches (e.g., JavaScript's big-number problem).

→ Don't pretend a remote service is a local object. Part of REST's appeal is treating network state transfer as explicitly not a function call.

Finding the service (service discovery & load balancing):

RPC evolution. Simplifying assumption: servers upgrade first, clients second → you need backward compatibility on requests, forward compatibility on responses. Compatibility is inherited from the encoding (gRPC/Protocol Buffers, Avro RPC; RESTful JSON where adding optional params/fields is compatible). Because RPC crosses organizational boundaries, you often can't force client upgrades → maintain compatibility a long time, sometimes running multiple API versions side by side. API versioning has no standard (URL version, Accept header, or per-API-key stored version).

Durable execution and workflows

A single business action (e.g., a payment) spans many service calls. A workflow is a graph of tasks (also called activities or durable functions), run by a workflow engine (an orchestrator that schedules + an executor that runs tasks).

Fullscreen

• Engines vary: ETL orchestration (Airflow, Dagster, Prefect), graphical BPMN (Camunda, Orkes), and durable execution (Temporal, Restate).
• Durable execution gives exactly-once semantics: you can't wrap two services in a DB transaction (and third-party gateways are outside your control), so the framework logs every RPC and state change to durable storage (like a WAL). On a retry it replays the workflow but skips already-completed calls, returning their prior results.
• Gotchas: external services must still offer idempotent APIs (use unique IDs); the framework expects the same RPCs in the same order, so reordering calls is brittle, deploy a new version rather than editing a running workflow; and replay must be deterministic, so avoid raw random/clock calls (use the framework's deterministic versions; tools like Temporal's Workflow Check catch nondeterminism). Determinism returns in Chapter 9.

Event-driven architectures

A request is an event/message; the sender usually doesn't wait, and the message goes through a message broker (event broker / message queue) that stores it temporarily.

Fullscreen

Advantages of a broker over direct RPC:

• Acts as a buffer if the recipient is down/overloaded (reliability).

• Redelivers to crashed consumers (no lost messages).

• No need for service discovery.

• Same message to multiple recipients.

• Decouples sender from recipient (sender doesn't care who consumes).

• Communication is asynchronous (send and forget; you can fake sync RPC with a reply channel).

• Brokers: RabbitMQ, ActiveMQ, NATS, Redpanda, Kafka; cloud Kinesis, Azure Service Bus, Pub/Sub (Ch. 12). Two patterns: queue (one consumer gets each message) and topic (all subscribers get it).

• A message is just bytes + metadata → use any encoding (Protocol Buffers/Avro/JSON + a schema registry; AsyncAPI is the messaging analog of OpenAPI). Many brokers delete messages after consumption; some keep them indefinitely (needed for event sourcing). If a consumer republishes, preserve unknown fields (Figure 5-1 again).

Distributed actor frameworks (Akka, Orleans, Erlang/OTP): the actor model encapsulates concurrency in actors with private state that communicate via async messages (one message at a time → no thread/locking worries). Distributed frameworks transparently encode messages across nodes. Location transparency works better here than in RPC because the actor model already assumes messages can be lost. But rolling upgrades still require forward/backward-compatible message encodings.

---

Real-world examples and analogies

• Backward vs forward compatibility, reading old vs future letters. Backward compatibility is reading a letter your grandmother wrote (you understand the old style). Forward compatibility is your grandmother reading a letter written in future slang, she must at least pass along the words she doesn't understand instead of crossing them out.
• The data-loss trap, editing a form with hidden fields. A new form has an extra field. Old software opens the form, can't see that field, saves it, and wipes the field it never knew about. Preserving unknown fields is "leave the parts you don't understand alone."
• Field tags (Protobuf) vs field names (JSON), seat numbers vs full names. Protobuf labels each value with a tiny seat number (tag) agreed in the schema, so the bytes stay compact and you can rename the person without re-issuing tickets. JSON writes everyone's full name on every ticket.
• Avro writer/reader schema, two editions of the same dictionary. The data is plain words with no labels; you can only read them with a dictionary. The writer used edition 3; you read with edition 5. Avro lines up the entries by headword (name), ignores words you dropped, and fills in defaults for new ones.
• Data outlives code, a filing cabinet you never re-file. You upgrade your label-maker (code) in an afternoon, but the decade-old folders keep their old labels until you physically re-file them. The system pretends everything uses today's labels by translating on read.
• RPC vs a phone call, the line can drop. A local function call is talking to someone in the same room. RPC is a phone call: it can drop mid-sentence, you may not know if they heard you, and re-saying it might place the order twice. Pretending it's the same as talking in-room (location transparency) causes bugs.
• Message broker, a post office. Instead of handing a letter directly to someone (who might be out), you drop it at the post office, which holds it, retries delivery, and can copy it to several recipients. You walk away immediately (async).
• Durable execution, a checklist you can resume. A long process (charge card, then deposit) keeps a checklist with results. If you faint halfway, you resume from the checklist instead of re-charging the card, as long as each step is idempotent and you don't reshuffle the steps.
• Idempotence + unique IDs, the "do not duplicate" key. Stamping each request with a unique ID is like a key marked "do not duplicate", even if the request is sent twice, the action happens once.

---

Cheat-sheet flashcards

Cover the answer and recall it.

• What is a rolling upgrade? → Deploying a new version to a few nodes at a time, no downtime.
• Backward compatibility? → New code can read old data.
• Forward compatibility? → Old code can read new data.
• Which is usually harder, and why? → Forward, old code must ignore/preserve new additions.
• What's the data-loss trap? → Old code rewrites a record and drops fields it didn't understand.
• Encoding = ? / Decoding = ? → In-memory → bytes / bytes → in-memory.
• Other names for encoding? → Serialization, marshaling.
• A zero-copy format example? → Cap'n Proto or FlatBuffers.
• Three problems with language-specific encodings? → Lock-in, security (arbitrary class instantiation), poor versioning/efficiency.
• Why do JSON numbers break in JavaScript? → Integers > 2^53 lose precision (IEEE 754 doubles).
• How do textual formats handle binary data? → Base64 (+~33% size).
• JSON Schema open vs closed content model? → Extra fields allowed (default) vs only defined fields.
• Are MessagePack/binary-JSON much smaller? → No, they still embed field names.
• What replaces field names in Protocol Buffers? → Field tags (numbers).
• Can you change a Protobuf field's tag? → No (breaks existing data); you can rename freely.
• Protobuf: add a field, how to stay compatible? → Give it a new tag; old code skips it; new code defaults missing ones.
• Protobuf: removing a field rule? → Never reuse its tag; reserve it.
• What's special about Avro's encoding? → No field names or tags, just concatenated values.
• Avro writer's vs reader's schema? → Schema used to write vs schema the reader expects (can differ).
• How does Avro match fields? → By name (schema resolution), order-independent.
• Avro rule for add/remove fields? → Only fields with a default value.
• How does an Avro reader learn the writer's schema? → Container-file header, version number + registry, or connection negotiation.
• Avro's advantage over Protobuf? → Friendlier to dynamically generated schemas (no manual tags).
• Two merits of binary schemas? → Compactness + always-current documentation/compatibility checks.
• "Data outlives code" means? → Old data stays in its original encoding long after the code is replaced.
• Best format for archival dumps? → Avro object container files or Parquet.
• What is a service/web service? → An application-specific API exposed over the network (HTTP).
• Two popular service IDLs? → OpenAPI/Swagger (JSON) and Protocol Buffers (gRPC).
• What is location transparency, and why is it flawed for RPC? → Making remote calls look local; networks fail, time out, vary, need idempotence.
• For RPC evolution, who upgrades first? → Servers; need backward-compatible requests, forward-compatible responses.
• Name two service-discovery approaches. → DNS, service-discovery system (etcd/ZooKeeper); also service mesh.
• What is durable execution? → Logging RPCs/state so a retried workflow skips completed steps (exactly-once).
• Why must durable-execution code be deterministic? → It replays; random/clock calls would diverge.
• Broker advantage over RPC (any two)? → Buffering, redelivery, no service discovery, fan-out, decoupling.
• Queue vs topic? → One consumer per message vs all subscribers get it.
• Why does location transparency work better for actors? → The model already assumes messages can be lost.

---

Common interview questions

1. Explain backward vs forward compatibility and why both matter during a rolling upgrade. (Old/new code coexist; new reads old + old reads new; needed so deploys don't break or lose data.)
2. Describe the forward-compatibility data-loss problem and how to avoid it. (Old code rewrites a record dropping unknown fields; preserve unknown fields.)
3. Compare JSON and a binary schema format like Protocol Buffers or Avro. (Human-readable/no schema/number ambiguity vs compact/schema-driven/typed/needs decoding.)
4. How does Protocol Buffers handle schema evolution? (Field tags; add with new tag, never reuse/change tags; defaults for missing fields; datatype-change truncation risk.)
5. How does Avro differ from Protocol Buffers, and what's the writer/reader schema? (No tags/names; match by name; resolution between writer and reader schemas; only add/remove fields with defaults; great for dynamic schemas.)
6. How does a database appear to use one schema when storage mixes versions ("data outlives code")? (Lazy migration; nullable columns filled on read; LSM rewrites on compaction.)
7. Why is RPC's location transparency a flawed abstraction? (Network failures, timeouts with unknown outcome, retries needing idempotence, variable latency, encoding params, language mismatches.)
8. What's the difference between synchronous RPC and a message broker, and when use each? (Wait-for-response vs fire-and-forget; broker buffers/redelivers/fans out/decouples.)
9. What is durable execution and how does it provide exactly-once semantics? (Log RPCs/state to a WAL; replay skips completed steps; requires idempotent external APIs + determinism.)
10. How do you handle service discovery and load balancing in a microservices system? (Hardware/software LB, DNS, discovery systems, service mesh, trade-offs.)
11. How do you version a public API when you can't force clients to upgrade? (Maintain multiple versions; URL/Accept header/per-key version; long-lived compatibility.)
12. Why does the actor model handle location transparency better than RPC? (Already assumes message loss and async; smaller local-vs-remote mismatch.)

---

Key terms glossary

---

Forward links (where these ideas return)

• Idempotence and exactly-once semantics → Chapter 12
• Network faults, timeouts, and determinism → Chapter 9
• Coordination services (etcd, ZooKeeper) → Chapter 10
• Event sourcing and message logs (Kafka) → Chapters 3 and 12
• Columnar archival (Parquet) and batch processing → Chapters 4 and 11
• Microservices and evolvability → Chapters 1 and 2