50 Protobuf JSON Marshal and Unmarshal

If you’ve ever worked with microservices, you’re surely already familiar with Protobuf as a serialization format. Protobuf’s support for fast yet schema-typed message exchange is the main reason this format is so popular in the Go ecosystem and many other languages. In practice, however, real-world demands often require JSON-based interoperability between services ― whether for logging, REST APIs, or debugging. This is exactly where Protobuf’s JSON marshal and unmarshal features become essential.

In this article, we’ll take a comprehensive look at the Marshal and Unmarshal JSON process for Protobuf in Go. We’ll explore the implementation, run through simulations, and cover the potential pitfalls you need to watch out for.

Terminology Summary

Why Is Protobuf JSON Marshal/Unmarshal Needed?

Even though Protobuf relies on binary encoding, many use cases still call for the JSON format. A few such situations include:

1. JSON-based logging for consistency across your stack.
2. REST APIs whose contracts require a JSON-formatted body.
3. Manual debugging of payloads.
4. Backward compatibility with legacy systems that still use JSON.

Process Flow Diagram

Let’s illustrate the flow using a simple flowchart diagram.

flowchart LR
    A[Struct Protobuf] --Marshal--> B[JSON]
    B --Unmarshal--> C[Struct Protobuf]

flowchart LR
    A[Struct Protobuf] --Marshal--> B[JSON]
    B --Unmarshal--> C[Struct Protobuf]

The flow above makes it clear that there’s a two-way process: from Protobuf to JSON (Marshal) and back again (Unmarshal).

Setting Up Protobuf and Go

Before diving into the code, make sure you have the following setup:

• Golang (1.18+)
• protoc
• The protobuf-go plugin: google.golang.org/protobuf

A simple example .proto file:

1syntax = "proto3";
2
3package article;
4
5message User {
6    int32 id = 1;
7    string name = 2;
8    bool is_active = 3;
9}

Generate the Go file:

1protoc --go_out=. user.proto

Implementing Marshal and Unmarshal

1. Marshal to JSON

google.golang.org/protobuf/encoding/protojson provides utilities for both Marshal and Unmarshal.

Example code:

 1package main
 2
 3import (
 4    "fmt"
 5    "log"
 6
 7    "google.golang.org/protobuf/encoding/protojson"
 8    pb "path/to/generated/package/article"
 9)
10
11func main() {
12    user := &pb.User{
13        Id:       50,
14        Name:     "Sophia",
15        IsActive: true,
16    }
17
18    // Marshal to JSON
19    data, err := protojson.Marshal(user)
20    if err != nil {
21        log.Fatalf("marshal error: %v", err)
22    }
23    fmt.Println(string(data))
24}

Output:

1{"id":50,"name":"Sophia","isActive":true}

Pro Tip:
By default, the output only includes fields that have been populated. For more readable output, use:

1marshaller := protojson.MarshalOptions{
2    Multiline:       true,
3    EmitUnpopulated: true,
4    Indent:          "  ",
5}
6jsonData, _ := marshaller.Marshal(user)
7fmt.Println(string(jsonData))

2. Unmarshal from JSON to Protobuf

Let’s simulate the reverse process: JSON to a Protobuf struct.

1jsonStr := `{"id":90,"name":"Aston","isActive":false}`
2
3var user pb.User
4err := protojson.Unmarshal([]byte(jsonStr), &user)
5if err != nil {
6    log.Fatalf("unmarshal error: %v", err)
7}
8fmt.Printf("Unmarshalled: %#v\n", user)

Output:

1Unmarshalled: article.User{Id:90, Name:"Aston", IsActive:false, ...}

Simulation: Handling Empty Fields

Below is a mapping table of marshal/unmarshal behavior regarding empty fields in Protobuf and JSON.

(1) Setting EmitUnpopulated: true in the Marshal options makes the JSON include default values.

Comparison Example:

1user := &pb.User{} // all defaults
2
3defaultOutput, _ := protojson.Marshal(user)
4withOpt := protojson.MarshalOptions{EmitUnpopulated: true}.Marshal(user)
5
6fmt.Println("Default:", string(defaultOutput))
7fmt.Println("EmitUnpopulated:", string(withOpt))

Output:

1Default: {}
2EmitUnpopulated: {"id":0,"name":"","isActive":false}

Pitfalls and Best Practices

1. Exposing Internal Fields

• Not every Protobuf field appears directly in JSON, such as oneof or map fields.

2. Default Values

• Default values in Protobuf (0, false, "") won’t automatically appear in JSON unless you set EmitUnpopulated.

3. Compatibility

• Protobuf JSON can differ from the JSON produced by ordinary Go struct tags. For example, field casing and enums may be written differently.

4. Arrays and Repeated Fields

• An empty list still appears as [] when EmitUnpopulated: true.

5. Oneof / Any

• Processing oneof and the Any type requires special attention. Sometimes it’s better to build a custom handler.

Advanced Scenario: Service Interop

Suppose one service sends JSON to another service, which then unmarshals it into Protobuf.

sequenceDiagram
    participant Client
    participant ServiceA
    participant ServiceB

    Client->>ServiceA: Kirim JSON Payload
    ServiceA->>ServiceA: Unmarshal ke Protobuf Struct
    ServiceA->>ServiceB: Marshal ke Binary Protobuf
    ServiceB->>ServiceB: Unmarshal dari Binary

sequenceDiagram
    participant Client
    participant ServiceA
    participant ServiceB

    Client->>ServiceA: Kirim JSON Payload
    ServiceA->>ServiceA: Unmarshal ke Protobuf Struct
    ServiceA->>ServiceB: Marshal ke Binary Protobuf
    ServiceB->>ServiceB: Unmarshal dari Binary

Conclusion

Marshaling and unmarshaling between Protobuf and JSON in Go is very straightforward with the help of protojson. However, like any meticulous engineer, it’s important to understand the default behavior, the available options, and the small antagonists such as empty fields or serialization distortions.

Work with a schema-first mindset and make sure your output has been validated against the consumers of each format. Experience will teach you the edge cases that aren’t always covered by standard tutorials.

If you’d like to discuss schema evolution and serialization in enterprise systems further, don’t hesitate to start a conversation in the comments. Happy building and happy marshalling! 🚀

References

• Go protobuf protojson package
• Official Protobuf Documentation
• Go Protocol Buffers Tutorial