67. Retry Mechanism in the gRPC Client: Building More Reliable Connections in Microservices

gRPC has become a popular communication foundation among modern microservices, both in cloud-native environments and conventional enterprise ecosystems. With features such as strong typing, contract-first APIs, streaming, and high performance built on the Protobuf protocol, gRPC is capable of handling a wide variety of service-to-service communication needs. But what happens when the connection between the client and the server is unstable? The answer: we need to apply a retry mechanism so that requests don’t fail easily and the application stays robust.

In this 67th article of our series on techniques for improving service availability, we’ll discuss how the retry mechanism in the gRPC client (in Go) works, why it matters, how to integrate it, and how to optimize it so it doesn’t become a “boomerang” in a distributed system.

Why Do We Need Retries in the Client?

In distributed systems, failures are normal—they can be caused by a slow network, a DNS timeout, a restarted server instance, or a BGP error. The retry mechanism is the first tool in an engineer’s “toolbox” for dealing with temporary failures (transient errors). The goal: maximize the chance of success without overburdening the server.

Table 1 – Example Errors and When to Retry

In short: retries are suitable for errors that are temporary in nature and usually resolve within a short time.

Retries in gRPC: Autopilot or Manual?

By default, the gRPC library does not automatically retry failed requests. The approaches are:

1. gRPC Retry Policy: A bit “magical”—since v1.8, gRPC has supported retry policies at the client level via the Service Config. Unfortunately, it is only fully implemented in the C++/Java/Python gRPC clients, while Go is still “experimental” (as of mid-2024) and the plugin is somewhat limited.
2. Manual Retry in the Client: In Go and TypeScript, for example, the common practice is to wrap the RPC call in your own retry block.

Here, we’ll focus on the manual approach so you understand what happens under the hood and can easily adapt it to your needs.

Basic Principles of Good Retries

A few guidelines so that our retries don’t become a new source of problems:

• Exponential backoff: Each time a call fails, the retry delay is extended. The goal: avoid flooding a server that is already struggling.
• Jitter: Add randomization to the delay so that a “stampede effect” doesn’t occur when many clients retry at the same time.
• Maximum number of attempts (Max attempts): Avoid endless retries.
• Idempotence-aware: Make sure the RPC method is idempotent to avoid double effects.

Illustration of the Retry Mechanism Flow

flowchart TD
    A[Start gRPC Call] --> B{RPC Return Error?}
    B -- No --> S[Success]
    B -- Yes --> C{Retryable Error?}
    C -- No --> F[Fail & Exit]
    C -- Yes --> D[Wait (Backoff + Jitter)]
    D --> E{Max Retry?}
    E -- No --> A
    E -- Yes --> F

flowchart TD
    A[Start gRPC Call] --> B{RPC Return Error?}
    B -- No --> S[Success]
    B -- Yes --> C{Retryable Error?}
    C -- No --> F[Fail & Exit]
    C -- Yes --> D[Wait (Backoff + Jitter)]
    D --> E{Max Retry?}
    E -- No --> A
    E -- Yes --> F

Case Study: Retry Mechanism for a gRPC Client in Go

Suppose we have a Go microservice that makes calls to a product catalog via gRPC. The GetProduct function is called by the API consumer, and sometimes an Unavailable error occurs because the catalog instance is under maintenance.

Let’s implement a simple retry mechanism:

1. Define the Retryable Error

 1func isRetryableError(err error) bool {
 2    if err == nil {
 3        return false
 4    }
 5    s, ok := status.FromError(err)
 6    if !ok {
 7        return false
 8    }
 9    switch s.Code() {
10    case codes.Unavailable, codes.DeadlineExceeded:
11        return true
12    default:
13        return false
14    }
15}

2. Retry Helper (Exponential Backoff + Jitter)

 1import (
 2    "math/rand"
 3    "time"
 4)
 5
 6func getBackoffDelay(attempt int, baseDelay time.Duration, maxDelay time.Duration) time.Duration {
 7    // Exponential backoff: baseDelay * 2^attempt
 8    delay := baseDelay * (1 << attempt)
 9    if delay > maxDelay {
10        delay = maxDelay
11    }
12    // Add jitter, for example up to 20% of the delay
13    jitter := time.Duration(rand.Int63n(int64(delay) / 5))
14    return delay + jitter
15}

3. Implementing the Retry Wrapper

 1func CallWithRetry(ctx context.Context, call func() error, maxAttempts int, baseDelay, maxDelay time.Duration) error {
 2    var err error
 3    for attempt := 0; attempt < maxAttempts; attempt++ {
 4        err = call()
 5        if !isRetryableError(err) {
 6            return err
 7        }
 8        if attempt < maxAttempts-1 {
 9            delay := getBackoffDelay(attempt, baseDelay, maxDelay)
10            // You can also check ctx.Done() here to abort if the context is cancelled
11            time.Sleep(delay)
12        }
13    }
14    return err
15}

4. Calling the RPC with Retry

 1maxAttempts := 5
 2baseDelay := 200 * time.Millisecond
 3maxDelay := 3 * time.Second
 4
 5err := CallWithRetry(ctx, func() error {
 6    // Call gRPC stub
 7    _, rpcErr := productClient.GetProduct(ctx, &pb.GetProductRequest{ProductId: "sku-123"})
 8    return rpcErr
 9}, maxAttempts, baseDelay, maxDelay)
10if err != nil {
11    log.Printf("failed GetProduct after retry: %v", err)
12}

Simulation: What Effect Do Retries Have on Traffic & Latency?

Suppose 1000 clients call the gRPC API at the same time, and 30% of the requests fail temporarily (timeout). With a maximum of 5 attempts, the total potential traffic is:

Total requests: Nearly 1,425 requests—an increase of 42.5% compared to no retries.

Be careful: if traffic is very high and nearly all requests error out, retries without limits can make the load worse (a thundering herd).

Optimization & Best Practices

• Use a context with a deadline: Retries are only performed within the time limit.
• Add tracing to retries: Log retry attempts and errors (with a correlation ID, for example).
• Combine with a circuit breaker: If the error rate is high, the circuit breaker opens temporarily and retries are not performed.
• Avoid retrying non-idempotent operations (inserts, money transfers, etc.).
• Configure retries on the client side as well as in the Service Config (when your language, library version, and infrastructure support it).

Conclusion

The retry mechanism in the gRPC client is an important strategy for making our service applications more resilient to temporary network or server failures. However, its implementation requires careful planning—avoid retrying blindly without limits, and make sure you only retry on the right errors and on operations that are safe to repeat. By adopting best practices such as exponential backoff, jitter, and traffic monitoring, an engineering team can prevent the system from becoming “overloaded” during downtime, while still delivering the best experience for end users.

Investing time in sensible retries will save hundreds of hours debugging production incidents.

Further Reading

• grpc-go Retry Support (experimental)
• Google Cloud: Best Practice for Retry
• Exponential Backoff and Jitter

Happy experimenting—and may the inter-service communication in your system grow ever more reliable! 🚀