108. Case Study: Automatically Generating Protobuf for Multiple Languages

Over the past decade, microservices have come to dominate backend system architecture. One of the keys to their success is the ability of various services to communicate with one another through a protocol that is consistent, fast, and yet easy to evolve. One popular format for this purpose is Protocol Buffers (Protobuf), introduced by Google.

In this article, I’ll share a real-world experience of how our team automated the process of generating Protobuf into code for several languages, in a consistent and efficient way. This case study combines best practices, the use of automation tools, CI/CD, and shows how the end result managed to reduce human error while speeding up service development.

1. The Initial Problem in a Cross-Stack Team

Our team consisted of Go, Node.js, and Java engineers. Every change to the Protobuf schema had to be followed by regenerating the code from the .proto files into each language binding. At first, this process was done manually:

• A developer submits changes to a .proto file
• Other backend teams pull the changes from the main branch
• They open a terminal and generate the code for their language (protoc-gen-go, protoc-gen-grpc-java, etc.)
• They commit the generated code into their respective service repositories

Unfortunately, this manual flow brought a number of problems:

1. Frequent version mismatches: One team forgets to regenerate, and another team ends up with code that has already diverged.
2. High potential for merge conflicts: Generated files often clashed.
3. Time-consuming: Especially for services that use many languages.
4. Hard to control tools and plugins: Generator and plugin versions often differed across developers’ machines.

Illustration of the Manual Flow

flowchart LR
    A[Update .proto files] --> B{Manual generate}
    B -- "Go" --> C[protoc-gen-go]
    B -- "Java" --> D[protoc-gen-grpc-java]
    B -- "Node.js" --> E[protoc-gen-ts]
    C --> F[Push ke repo Go]
    D --> G[Push ke repo Java]
    E --> H[Push ke repo Node.js]

flowchart LR
    A[Update .proto files] --> B{Manual generate}
    B -- "Go" --> C[protoc-gen-go]
    B -- "Java" --> D[protoc-gen-grpc-java]
    B -- "Node.js" --> E[protoc-gen-ts]
    C --> F[Push ke repo Go]
    D --> G[Push ke repo Java]
    E --> H[Push ke repo Node.js]

2. The Solution: Automating Multi-Language Protobuf Generation

To address the problems above, we decided to automate the entire code generation process in our CI/CD system (GitHub Actions), and to create a shared proto repository. The new flow works like this:

• All .proto files are stored in a single repo
• On every commit or PR, GitHub Actions automatically generates the language bindings while checking for compatibility
• The generated files are committed to a separate branch, or uploaded as artifacts
• The Go, Node.js, and Java services simply pull the latest release or the generated code package

High-Level Diagram

flowchart TD
    RepoProto["Shared Proto Repo (.proto)"]
    CI["CI/CD Workflow (GitHub Actions)"]
    GoPkg["Go SDK Artifacts / Package"]
    JavaPkg["Java SDK Artifacts / Package"]
    NodePkg["Node.js SDK Artifacts / Package"]
    ServiceA["Go Service (import paket)"]
    ServiceB["Node.js Service (import paket)"]
    ServiceC["Java Service (import paket)"]

    RepoProto --> CI
    CI --> GoPkg
    CI --> JavaPkg
    CI --> NodePkg
    GoPkg --> ServiceA
    JavaPkg --> ServiceC
    NodePkg --> ServiceB

flowchart TD
    RepoProto["Shared Proto Repo (.proto)"]
    CI["CI/CD Workflow (GitHub Actions)"]
    GoPkg["Go SDK Artifacts / Package"]
    JavaPkg["Java SDK Artifacts / Package"]
    NodePkg["Node.js SDK Artifacts / Package"]
    ServiceA["Go Service (import paket)"]
    ServiceB["Node.js Service (import paket)"]
    ServiceC["Java Service (import paket)"]

    RepoProto --> CI
    CI --> GoPkg
    CI --> JavaPkg
    CI --> NodePkg
    GoPkg --> ServiceA
    JavaPkg --> ServiceC
    NodePkg --> ServiceB

3. Step-by-Step: Building an Automated Protobuf Pipeline

Let’s break down each stage, complete with example configurations.

3.1 Repository Folder Structure

 1proto-shared/
 2├── protos/
 3│   ├── example/
 4│   │   ├── hello.proto
 5├── build/
 6│   ├── go/
 7│   ├── java/
 8│   ├── node/
 9├── .github/
10│   └── workflows/
11│       └── generate.yml

3.2. Example .proto File

 1// protos/example/hello.proto
 2syntax = "proto3";
 3
 4package example;
 5
 6service Greeter {
 7  rpc SayHello (HelloRequest) returns (HelloReply) {}
 8}
 9
10message HelloRequest {
11  string name = 1;
12}
13
14message HelloReply {
15  string message = 1;
16}

3.3. CI/CD Workflow Definition (GitHub Actions)

Inside .github/workflows/generate.yml:

 1name: Generate Protobuf for Multiple Languages
 2
 3on:
 4  push:
 5    branches: [ main ]
 6  pull_request:
 7    branches: [ main ]
 8
 9jobs:
10  build:
11    runs-on: ubuntu-latest
12
13    steps:
14      - uses: actions/checkout@v3
15      - name: Setup Go
16        uses: actions/setup-go@v4
17        with:
18          go-version: '1.21.0'
19      - name: Setup Node
20        uses: actions/setup-node@v3
21        with:
22          node-version: '18.x'
23      - name: Setup Java
24        uses: actions/setup-java@v3
25        with:
26          distribution: 'temurin'
27          java-version: '17'
28
29      - name: Install Protobuf Compiler
30        run: sudo apt-get install -y protobuf-compiler
31
32      - name: Install protoc plugins
33        run: |
34          go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
35          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
36          npm install -g protoc-gen-ts
37
38      - name: Generate Go code
39        run: |
40          mkdir -p build/go
41          protoc -I=protos --go_out=build/go --go-grpc_out=build/go protos/example/*.proto
42
43      - name: Generate Node.js code
44        run: |
45          mkdir -p build/node
46          protoc -I=protos --js_out=import_style=commonjs:build/node --grpc_out=build/node protos/example/*.proto
47
48      - name: Generate Java code
49        run: |
50          mkdir -p build/java
51          protoc -I=protos --java_out=build/java --grpc-java_out=build/java protos/example/*.proto
52
53      - name: Upload Build Artifacts
54        uses: actions/upload-artifact@v3
55        with:
56          name: generated-protobuf
57          path: build/

Brief Explanation:

• Set up the environment for all three languages in a single workflow
• Install the protocol compiler & plugins as needed for multi-language support
• Generate the source code for Go, Node.js, and Java into separate build folders
• Upload artifacts that each team/service can retrieve

4. Consuming the Generated Output: A Go Service Simulation

For the Go team, they simply download the artifacts from CI/CD or fetch them from a release/package. Here’s a simulation of how it’s used:

 1import (
 2    "context"
 3    "log"
 4    pb "github.com/org/proto-shared/build/go/example"
 5)
 6
 7func main() {
 8    conn, err := grpc.Dial("greeter-service:50051", grpc.WithInsecure())
 9    if err != nil {
10       log.Fatal(err)
11    }
12    defer conn.Close()
13    client := pb.NewGreeterClient(conn)
14    resp, err := client.SayHello(context.Background(), &pb.HelloRequest{Name: "Budi"})
15    if err != nil {
16        log.Fatal(err)
17    }
18    log.Println("Received:", resp.Message)
19}

5. Comparison Table: Before vs. After Automation

6. Lessons Learned & Implementation Tips

Lessons Learned:

• A centralized repository for .proto files makes versioning far easier
• CI/CD must be automated so that no manual steps remain
• Build artifacts are better than committing generated output directly, to avoid merge conflicts

Implementation Tips:

• If you already have a monorepo, use submodules for teams that have their own repos.
• Use semantic versioning for .proto releases
• Also build a docker image containing the result if it needs to be used in other containers.

7. Conclusion

Automating Protobuf generation for multiple languages has been a game-changer for our team’s development process. What used to be error-prone and time-consuming is now elegant, scalable, and easy to integrate into any pipeline.

If you work on a multi-stack team, don’t hesitate to invest the time in building this foundation! The long-term benefits are real — faster feature releases, fewer errors, and far more seamless integration between teams.

Feel free to discuss further if you’d like to know the setup details or need a more advanced script!