48 Custom Options in Protobuf

If you’ve ever worked with Google’s Protobuf (Protocol Buffers), you surely know how powerful this serialization tool is for cross-platform and cross-language applications. But did you know that Protobuf isn’t just about defining simple data structures? There’s a feature that often gets overlooked yet is incredibly powerful—namely Custom Options. This time I’ll dig into this feature in depth, including code examples, scenario simulations, and a few diagrams to make the flow clearer.

A Quick Introduction to Protobuf and Options

By default, Protobuf already provides a wide range of built-in options, such as deprecated, packed, and so on. These are used to attach additional metadata to fields, messages, enums, and the like.

For example:

1message Person {
2  optional string name = 1 [deprecated=true];
3}

You can set [deprecated=true] to mark a field that should no longer be used.

But sometimes developers need more than just standard metadata. What if we want to store specific information, for example: “this particular field must be encrypted at the application level”, or to mark “this field should appear in the REST API documentation”, and so on? This is where Custom Options come in as the solution.

What Are Custom Options?

Custom Options give us a way to define new (custom) metadata on almost any element in a .proto file: fields, messages, enums, services, methods—depending on what you need. Custom options are created by declaring an extension against one of several reserved option holders, such as google.protobuf.FieldOptions.

Why Do We Need Custom Options?

• Additional metadata that can be processed at both build time and runtime.
• Integration with internal tools (e.g., custom code generators, internal documentation tools, validators, etc.).
• Process automation, such as testing, logging, and even access control.

How Custom Options Work

To make use of custom options, there are 3 main steps:

1. Create the Custom Option Definition.
2. Use the Custom Option in another .proto.
3. Access the Custom Option at build time or runtime.

Let’s break down each step with real code.

1. Create the Custom Option Definition

A custom option is defined using the extension feature in .proto, applied to one of Google’s special google.protobuf.*Options messages.

For example, say we want to provide the information “this field must be exported to the API” via the api_exported option at the field level:

1// custom_options.proto
2import "google/protobuf/descriptor.proto";
3
4extend google.protobuf.FieldOptions {
5  optional bool api_exported = 50001;
6}

2. Use the Custom Option in Another .proto

Once the extension is declared, we then import that extension into the relevant .proto file.

1// person.proto
2import "custom_options.proto";
3
4message Person {
5  string name = 1 [(api_exported) = true];
6  int32 age = 2; // without the option
7}

The name field now has new metadata: (api_exported) = true.

3. Reading Custom Options

To read a custom option, you need support from the tools/language you’re using. The Protobuf libraries for Java, Go, C++, and Python generally already support accessing these options via reflection / the protobuf API.

Example: Reading Custom Options in Python

1# install protobuf: pip install protobuf
2from person_pb2 import Person, DESCRIPTOR
3from custom_options_pb2 import api_exported
4
5field = DESCRIPTOR.fields_by_name['name']
6if field.GetOptions().Extensions[api_exported]:
7    print("Field 'name' is exported to API")

Use Cases & Case Studies

Use Case 1: Field-level Data Masking

Suppose there’s a security requirement: “Every field tagged with the custom mask = true option must be automatically masked during serialization!”

Extension definition:

1// masking.proto
2import "google/protobuf/descriptor.proto";
3extend google.protobuf.FieldOptions {
4  optional bool mask = 50011;
5}

Usage:

1// customer.proto
2import "masking.proto";
3
4message Customer {
5  string name = 1;
6  string phone = 2 [(mask) = true];
7}

Runtime logic (Go-like pseudocode):

1for _, field := range proto.MessageFields(customer) {
2  if field.Options().GetExtension(mask) == true {
3    MaskField(&customer, field)
4  }
5}

Use Case 2: Automatic Endpoint Documentation

With a custom option, a note like “This endpoint is a beta version” can be embedded directly into the .proto and used by an internal documentation generator.

Extension definition:

1// doc_tag.proto
2import "google/protobuf/descriptor.proto";
3extend google.protobuf.MethodOptions {
4  optional string doc_tag = 50021;
5}

Usage:

1service UserService {
2  rpc GetUser(UserRequest) returns (User) {
3    option (doc_tag) = "beta";
4  }
5}

Visualizing the Protobuf Compilation Process With Custom Options

Let’s see how custom options flow from the .proto to the final result through a mermaid diagram.

graph LR
    A[.proto dengan Custom Option] --> B[protoc compiler]
    B --> C["File hasil compile: *_pb2(.py|.go|.java)"]
    C --> D[Code Generation / Reflection API]
    D --> E[Read/Process Custom Metadata]

graph LR
    A[.proto dengan Custom Option] --> B[protoc compiler]
    B --> C["File hasil compile: *_pb2(.py|.go|.java)"]
    C --> D[Code Generation / Reflection API]
    D --> E[Read/Process Custom Metadata]

• Node A: The developer writes a .proto with custom options.
• Node B: protoc compiles the .proto (embedding the custom extension in the descriptor file).
• Node C: The output is the code/descriptor file that stores the custom option information.
• Node D: A tool or library reads the descriptor to detect the custom options.
• Node E: Custom tools/processes can access, process, or react based on the value of the custom options.

Table: Where Custom Options Can Be Applied

Build Simulation: Step-by-step

Suppose you have three files: custom_options.proto, person.proto, and you then compile them:

1protoc --python_out=. custom_options.proto person.proto

If you have a custom tool (e.g., a validator), the pipeline roughly looks like this:

flowchart LR
    subgraph Developer
    F1[Edit .proto]
    end
    subgraph Build
    F2[protoc compile]
    F3[Custom tool analyze pb2]
    end
    F1 --> F2 --> F3

flowchart LR
    subgraph Developer
    F1[Edit .proto]
    end
    subgraph Build
    F2[protoc compile]
    F3[Custom tool analyze pb2]
    end
    F1 --> F2 --> F3

Limitations & Important Notes

1. Custom options are only metadata; they don’t change the serialization format/binary.
2. Not every code generator/language binding automatically exposes custom options.
3. Options can only be read via reflection/descriptors.

Conclusion

Custom options in Protobuf are a magical yet underrated feature that can save a lot of time, maintain consistency, and enrich the metadata of a developer’s workflow. With a little setup, you can build a more maintainable and powerful ecosystem, especially for organizations whose workflows rely heavily on Protobuf.

If you’re building your own tooling, or want to enrich your data models and APIs, don’t forget to review and explore this feature.

Have a unique experience using custom options? Share it in the comments!