68. Data Compression (gzip) in gRPC: Optimizing Microservice Performance

In the era of cloud-native and microservices, the efficiency of communication between services becomes increasingly vital. One thing that often turns into a bottleneck is bloated data payloads, especially over congested networks with high latency. One elegant solution is data compression. In this article, we’ll discuss how to implement data compression using gzip in gRPC, as well as its impact on service performance.

Why Is Data Compression in gRPC Important?

Communication between services—whether through HTTP/gRPC REST or protocol-buffer-based RPC—is prone to becoming a performance bottleneck if the data payload being sent is too large. This can lead to:

• Low throughput because a large amount of bandwidth is consumed.
• High latency because data takes a long time to transmit.
• Increased cost in cloud environments that charge per bandwidth.

gRPC, as a modern RPC framework, is fortunate to already include support for payload compression, including gzip.

How Does gRPC Compression Work?

Fundamentally, gRPC allows the client and server to negotiate the encoding of the data. By default, gRPC sends data uncompressed. However, we can specify an encoding such as gzip—commonly used because it’s widely supported across many programming languages.

Here is a request-response flow diagram with data compression enabled:

sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: (Request dengan gzip)
    S-->>C: (Response dikompresi gzip)

sequenceDiagram
    participant C as Client
    participant S as Server

    C->>S: (Request dengan gzip)
    S-->>C: (Response dikompresi gzip)

Illustrating the Problem and Explaining the Case

Imagine that within a microservice setup, service A calls service B to fetch a large list of transactions (for example, a customer’s transaction history over a 1-year period). The payload being sent could reach hundreds of KB or even several MB. If the network connection is limited—say, between data centers in different regions—then the transfer time becomes significant.

Payload Simulation

Configuring Data Compression (gzip) in gRPC

Let’s look at how it’s applied on the Go side. The implementation in other languages such as Python or Java follows a similar pattern.

1. Enabling Compression on the Client

Here’s a simple example of gRPC client code with gzip compression:

 1import (
 2    "context"
 3    "google.golang.org/grpc"
 4    "google.golang.org/grpc/encoding/gzip"
 5    pb "myapp/proto"
 6)
 7
 8func main() {
 9    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
10    if err != nil {
11        log.Fatalf("did not connect: %v", err)
12    }
13    defer conn.Close()
14
15    client := pb.NewMyServiceClient(conn)
16
17    resp, err := client.ListTransactions(
18        context.Background(),
19        &pb.ListTransactionsRequest{
20            UserId: "user-0042",
21        },
22        grpc.UseCompressor(gzip.Name), // <--- Enable gzip compression
23    )
24    if err != nil {
25        log.Fatalf("Error during call: %v", err)
26    }
27    log.Printf("Got response: %v", resp)
28}

2. Supporting Compression on the Server

So that the server can accept and respond to compressed requests, add the following options when registering the server:

 1server := grpc.NewServer(
 2    grpc.RPCCompressor(grpc.NewGZIPCompressor()), // For grpc <=v1.8 (deprecated)
 3)
 4
 5server := grpc.NewServer(
 6    grpc.Creds(...), // TLS settings, etc.
 7    grpc.StreamInterceptor(...),
 8    grpc.UnaryInterceptor(...),
 9    grpc.WriteBufferSize(1024*1024), // Tuning
10    grpc.ReadBufferSize(1024*1024),
11)

3. Enabling Automatic Compression/Decompression in gRPC (All Languages)

Most gRPC implementations provide a flag to force or auto-select compression in the header. In Python:

 1import grpc
 2from grpc import gzip_compression
 3
 4channel = grpc.insecure_channel('localhost:50051')
 5stub = MyServiceStub(channel)
 6
 7response = stub.ListTransactions(
 8    ListTransactionsRequest(user_id='user-0042'), 
 9    compression=grpc.Compression.Gzip  # Enabling compression
10)

Testing Compression Effectiveness

Let’s prove its effectiveness. I’m using 2 data samples: uncompressed JSON vs. compressed gRPC protobuf.

Request Latency Simulation

Test results from a dev environment with a 10 Mbps connection, 10k records.

Disclaimer: The CPU overhead for compression/decompression needs to be taken into account, but on modern servers it generally doesn’t become a bottleneck compared to network transfer time.

gRPC Negotiation Standard

gRPC will automatically send the following headers to the server:

1grpc-encoding: gzip
2accept-encoding: gzip,identity

If the server supports gzip, the response is automatically compressed in accordance with the request. If not, the response falls back to the default encoding (identity/uncompressed).

When Should Compression Be Enabled?

• Large payload size: Compression is very helpful when records exceed 50–100 KB.
• Limited bandwidth or high latency: On inter-region/global networks.
• Microservice Pattern: When data exchange between services is very intense and payloads are large, optimization is a must.

However: For small request-response pairs (< 4 KB), the CPU overhead of compression outweighs the benefits—just stick with uncompressed!

Challenges and Best Practices

Client Side

• Use compression only when needed (large payloads)
• Make sure the client and server use libraries/languages with comparable support

Server Side

• Monitor CPU consumption (compression requires resources)
• Set a maximum payload limit, with a fail-safe for abnormal data
• Always handle the negotiation header (compatibility with older API clients)

Case Study

For example, in a banking backend system:

• Without compression: 1 million queries/day x 1 MB => 1 TB/day of traffic
• With compression (80%): 1 million x 200 KB => 200 GB/day

That saves 800 GB of data per day, which translates into both performance gains and lower bandwidth costs!

Conclusion

Data compression (gzip) in gRPC is a simple yet powerful feature. It not only reduces traffic costs, but also lowers latency, speeds up responses, and improves the user experience. With just one or two extra lines of code, our microservices become far more scalable in distributed environments.

Don’t forget to always measure before and after enabling compression in a production environment to balance CPU resources against data transfer efficiency.

References & Further Reading

• gRPC: Compression
• Protobuf Size Efficiency
• gRPC Go Docs: Compression

See you in the next microservice experiment article. Always measure, optimize, and be ready to scale up! 🚀