41 Understanding Nested Messages

Protocol Buffers (protobuf) is one of the most powerful serialization formats and is widely used in the development of distributed systems or applications that require inter-service communication. Behind its efficiency and flexibility, one feature that often raises questions among both new and senior engineers is nested messages.

In this 41st article of the Protocol Buffers series, I’ll break down the concept of nested messages from its definition and implementation all the way to the best practices I’ve discovered over years of managing data models with protobuf. We’ll walk through the rules, work through edge cases, and test the validity and flexibility of nested messages using code examples, flow diagrams, and real-world simulations. Let’s get started!

What Are Nested Messages?

Put simply, a nested message in protobuf is a message defined inside another message. You can think of it like an inner class in Java or a struct within a struct in C. Nested messages are extremely useful for grouping closely related data and avoiding pollution of the global namespace.

Basic Structure

Let’s look at the following basic example:

 1syntax = "proto3";
 2
 3message AddressBook {
 4  message Person {
 5    string name = 1;
 6    int32 id = 2;
 7    string email = 3;
 8  }
 9  repeated Person people = 1;
10}

Explanation:

• AddressBook is the main message.
• Person is a nested message inside AddressBook.
• The people field will contain a list of Person objects.

When Should You Use Nested Messages?

In my experience, nested messages are particularly useful when:

• The relationship between data is hierarchical or one-directional.
• The message structure is not used globally or across files.
• You want to organize data types that are only relevant to a specific message.

Table: Comparing the Use of Nested vs. Top-level Messages

Accessing Nested Messages in Code

Generate the Protobuf File

1protoc --go_out=. --go-grpc_out=. addressbook.proto

A nested message implementation automatically produces an inner class (in Java/Python) or a nested namespace in Go/C++. Here are some examples:

Java

1AddressBook.Person person = AddressBook.Person.newBuilder()
2    .setName("Luna")
3    .setId(1)
4    .setEmail("luna@example.com")
5    .build();

Go

1person := &pb.AddressBook_Person{
2    Name:  "Luna",
3    Id:    1,
4    Email: "luna@example.com",
5}

Note: Notice how in Go, a nested message produces a type named Parent_Child (AddressBook_Person).

Simulation: Serializing and Deserializing Nested Messages

Let’s try a simple data serialization-deserialization simulation using Python.

1. Protofile: addressbook.proto

 1syntax = "proto3";
 2package example;
 3
 4message AddressBook {
 5  message Person {
 6    string name = 1;
 7    int32 id = 2;
 8    string email = 3;
 9  }
10  repeated Person people = 1;
11}

2. Generate the Go File from the Proto

Run the following command to generate the Go file (make sure you have installed protoc and the Go plugin):

1protoc --go_out=. addressbook.proto

Output: addressbook.pb.go

3. Go Code to Serialize & Deserialize

 1package main
 2
 3import (
 4	"fmt"
 5	"log"
 6
 7	"google.golang.org/protobuf/proto"
 8
 9	// Replace with the package path of your generated proto
10	examplepb "path/to/your/generated/example"
11)
12
13func main() {
14	// Create an AddressBook and add 1 Person
15	book := &examplepb.AddressBook{
16		People: []*examplepb.AddressBook_Person{
17			{
18				Name:  "Lenka",
19				Id:    42,
20				Email: "lenka@medium.com",
21			},
22		},
23	}
24
25	// Serialize to []byte
26	data, err := proto.Marshal(book)
27	if err != nil {
28		log.Fatalf("Failed to serialize: %v", err)
29	}
30
31	// Deserialize from []byte
32	parsed := &examplepb.AddressBook{}
33	if err := proto.Unmarshal(data, parsed); err != nil {
34		log.Fatalf("Failed to parse: %v", err)
35	}
36
37	// Print the result
38	for _, person := range parsed.People {
39		fmt.Println("Name :", person.Name)
40		fmt.Println("ID   :", person.Id)
41		fmt.Println("Email:", person.Email)
42	}
43}

🧾 Output:

1Name : Lenka
2ID   : 42
3Email: lenka@medium.com

✅ Notes:

• Replace examplepb with the Go module path from the generated addressbook.pb.go.
• Use a module path like github.com/username/project/proto/example if you are using a Go module.

Flow Diagram: The Nested Message Serialization Process

We can visualize the serialization and deserialization process of a nested message with the following diagram (using Mermaid):

flowchart LR
  A[Create AddressBook] --> B[Add Person]
  B --> C[Set Attributes]
  C --> D[Serialize To Bytes]
  D --> E[Deserialize From Bytes]
  E --> F[Read Nested Person Data]

flowchart LR
  A[Create AddressBook] --> B[Add Person]
  B --> C[Set Attributes]
  C --> D[Serialize To Bytes]
  D --> E[Deserialize From Bytes]
  E --> F[Read Nested Person Data]

Edge Cases and Practical Tricks

Here are some edge cases and best practices around nested messages that I frequently encounter:

1. Forward References Are Not Allowed

A nested message cannot reference its parent message directly, because the parent has not yet been fully defined at parse time. Use a field of a different type or refactor the message if two-way dependencies are required.

2. Using Repeated Nested Messages

Add a repeated field for an array of nested messages:

1repeated Person people = 1;

3. Going Deeper? Be Careful!

Protobuf does support nesting to deep levels, but going too deep reduces readability and increases serialization complexity. My recommendation: a maximum of two levels whenever possible.

4. Reusing a Nested Message Outside Its Parent

If you have to use the same type elsewhere, move the nested message declaration to the top level so it can be imported and referenced by other messages.

Pros & Cons of Nested Messages

A Simple Case Study: Payment Log

Suppose we’re building a payment logging system:

1message PaymentLog {
2  string transaction_id = 1;
3  message ItemDetail {
4    string item_id = 1;
5    int32 quantity = 2;
6  }
7  repeated ItemDetail items = 2;
8  double total = 3;
9}

With this design, the ItemDetail type can only be used in the context of a payment transaction, and won’t clash when you have a similar message with the same fields in a different context.

Conclusion

Nested messages are a powerful feature of Protocol Buffers that helps us maintain the structure and modularity of our data models. The key is understanding the context in which they’re used and their potential technical implications. As an engineer, use nested messages to:

• Organize data types.
• Prevent pollution of the global namespace.
• Limit the scope of types that don’t need to be exposed externally.

That said, be mindful of reusability and maintainability! Use nested messages wisely, in line with the principles of encapsulation and separation of concerns.

We’ve walked through the theory, hands-on simulations, and the tendencies I’ve encountered in the field. If you have any experience or questions about nested messages in protobuf, feel free to share them in the comments! 🚀

Additional References

• Official Protocol Buffers Guide
• Best Practices in Protobuf Design (Uber Engineering)

Thanks for reading. Don’t forget to follow the next installment in the series!