46 Protobuf Schema Compatibility and Evolution

Introduction

Transmitting data between microservices has become a pillar of modern backend architecture. One of the popular technologies used for this is Protocol Buffers (Protobuf) . Its main advantages include fast serialization, minimal payload size, cross-language support, and—just as importantly—flexibility in schema evolution. However, maintaining compatibility as a schema evolves can become complex and fragile if we are not careful. This article thoroughly explores that important aspect, from theory and practice to code examples and simulations.

What Is Schema Compatibility?

Put simply, schema compatibility means that an older version of an application is still able to read or write data using a newer version of the schema (or vice versa) without errors that would corrupt data integrity.

In Protobuf, there are two important terms:

• Backward Compatibility: Old applications can read data produced by new applications.
• Forward Compatibility: New applications can read data produced by old applications.

The key in Protobuf is the use of explicit field tags on every field. This is what distinguishes Protobuf from formats such as JSON or XML.

A Simple Schema Example

1// v1: Initial Message
2message UserProfile {
3  string name = 1;
4  int32 age = 2;
5}

Schema Evolution: A Case Study

Imagine you need to add a new email field. What should you keep in mind to stay compatible?

1// v2: email added
2message UserProfile {
3  string name = 1;
4  int32 age = 2;
5  string email = 3; // add a new field with a unique tag
6}

Explanation:

• The old fields (name, age) keep their old tags (1, 2).
• The new field (email) uses a new tag (3).

An old application that only knows about v1 will automatically ignore field tag 3 and discard it, so both backward and forward compatibility remain intact.

The Golden Rules of Protobuf Schema Evolution

Here is a quick table of schema evolution actions and their impact on compatibility:

Flow Diagram: Recommended Schema Evolution

flowchart TD
    A[Mulai Evolusi Schema]
    B{Perlu Tambah Field?}
    C{Perlu Hapus Field?}
    D[Re-use tag lama?]
    E[Aman: tambahkan dengan tag baru]
    F[Aman: biarkan tag menganggur]
    G[Berbahaya: jangan reuse tag!]
    H[Perlu Ubah Type Field?]
    I[Berbahaya: akan korup data!]
    J[Selesai]

    A --> B
    B -- ya --> E
    B -- tidak --> C
    C -- ya --> D
    D -- ya --> G
    D -- tidak --> F
    C -- tidak --> H
    H -- ya --> I
    H -- tidak --> J
    E --> J
    F --> J
    G --> J
    I --> J

flowchart TD
    A[Mulai Evolusi Schema]
    B{Perlu Tambah Field?}
    C{Perlu Hapus Field?}
    D[Re-use tag lama?]
    E[Aman: tambahkan dengan tag baru]
    F[Aman: biarkan tag menganggur]
    G[Berbahaya: jangan reuse tag!]
    H[Perlu Ubah Type Field?]
    I[Berbahaya: akan korup data!]
    J[Selesai]

    A --> B
    B -- ya --> E
    B -- tidak --> C
    C -- ya --> D
    D -- ya --> G
    D -- tidak --> F
    C -- tidak --> H
    H -- ya --> I
    H -- tidak --> J
    E --> J
    F --> J
    G --> J
    I --> J

Evolution Simulation: A Code Study

Below is the Golang version of a two-way example demonstrating both forward compatibility and backward compatibility in Protocol Buffers:

user_v1.proto

1syntax = "proto3";
2package userpb;
3
4message UserProfile {
5  string name = 1;
6  int32 age = 2;
7}

user_v2.proto

1syntax = "proto3";
2package userpb;
3
4message UserProfile {
5  string name = 1;
6  int32 age = 2;
7  string email = 3;
8}

⚙️ Generating the Go Files

1protoc --go_out=. user_v1.proto
2protoc --go_out=. user_v2.proto

This will produce:

• user_v1.pb.go
• user_v2.pb.go

Go Code: Simulating Two-Way Compatibility

 1package main
 2
 3import (
 4	"fmt"
 5	"log"
 6
 7	"google.golang.org/protobuf/proto"
 8
 9	userpbv1 "path/to/user_v1"
10	userpbv2 "path/to/user_v2"
11)
12
13func main() {
14	fmt.Println("=== 1. Old Version Reading New Version Data ===")
15
16	// V2 writes data with an additional field: email
17	msgV2 := &userpbv2.UserProfile{
18		Name:  "Budi",
19		Age:   30,
20		Email: "budi@example.com",
21	}
22	data, err := proto.Marshal(msgV2)
23	if err != nil {
24		log.Fatalf("Failed to serialize v2: %v", err)
25	}
26
27	// Read by the old version (v1) that does not know about the email field
28	msgV1 := &userpbv1.UserProfile{}
29	if err := proto.Unmarshal(data, msgV1); err != nil {
30		log.Fatalf("Failed to deserialize v1: %v", err)
31	}
32	fmt.Println("Deserialized (v1):", msgV1) // email is ignored without error
33
34	fmt.Println("\n=== 2. New Version Reading Old Version Data ===")
35
36	// V1 writes data without the email field
37	msgV1Old := &userpbv1.UserProfile{
38		Name: "Rina",
39		Age:  20,
40	}
41	dataOld, err := proto.Marshal(msgV1Old)
42	if err != nil {
43		log.Fatalf("Failed to serialize v1: %v", err)
44	}
45
46	// Read by the new version (v2) that expects email
47	msgV2Read := &userpbv2.UserProfile{}
48	if err := proto.Unmarshal(dataOld, msgV2Read); err != nil {
49		log.Fatalf("Failed to deserialize v2: %v", err)
50	}
51	fmt.Println("Deserialized (v2):", msgV2Read) // email == "", default
52}

Output:

1=== 1. Old Version Reading New Version Data ===
2Deserialized (v1): name:"Budi" age:30
3
4=== 2. New Version Reading Old Version Data ===
5Deserialized (v2): name:"Rina" age:20 email:""

Result:

The Worst Practice: Reusing a Tag Number

A scenario you must avoid:

1// v2 - WRONG, reusing a tag!
2message UserProfile {
3  string name = 1;
4  int32 age = 2;
5  string phone = 3; // was 'email', now 'phone', with the same tag!
6}

Parsing old data can lead to field corruption. Data under tag 3 may contain an email, but it gets interpreted as a phone number.

Engineer Tips: Safe Evolution

1. Never Change a Field’s Type
   Changing a field’s type (for example, from string to int32) can potentially cause parsing errors and data corruption.

2. Never Reuse a Tag Number
   Once a tag is removed, leave it unused forever.

3. Avoid Changing Optional to Required
   Old messages that lack the field will fail to parse if it becomes mandatory.

4. Consider Versioning
   For major changes, use a new message: UserProfileV2 and so on, and create a data migration procedure.

5. Read the Official Protobuf Evolution Guide

Conclusion

Schema compatibility is a vital aspect of integrating Protobuf-based systems. By following a few simple rules (never change tag numbers or field types, and always add fields additively), we can ensure schema evolution proceeds without drama. The simulation above shows how backward and forward compatibility work in practice.

Always test your schema compatibility whenever changes occur and, if possible, build a CI pipeline so that this schema integrity is always preserved. Evolving fast, but without breaking things.

References:

• https://developers.google.com/protocol-buffers/docs/proto3#updating
• https://protobuf.dev/programming-guides/proto3/

Happy engineering! 🚀