64. Implementing a gRPC Health Server

gRPC has become one of the most popular communication protocols in modern distributed systems development. Beyond its high performance and multi-language support, gRPC also boasts a rich ecosystem, one part of which is its health checking mechanism. In this article, I’ll take an in-depth look at implementing a gRPC Health Server, why we need it, and how to integrate it into your microservices. I’ll also include code examples, simulations, and flow diagrams so you can put these concepts into practice right away.

Why Do We Need a Health Server in gRPC?

In the world of microservices, service health monitoring is absolutely crucial. A service mesh like Istio or an orchestrator like Kubernetes needs a standard way to determine whether our service is “alive” and “ready to accept traffic.” Without a standardized health check mechanism, automated processes such as rolling updates, load balancing, and automated recovery can fail.

From the very beginning, gRPC did not provide health checking by default. However, the Cloud Native Computing Foundation (CNCF), through grpc-ecosystem/grpc-health-probe , introduced the Health Checking Protocol as an additional extension. This health server speaks the same protocol (protobuf/gRPC), making the implementation and consumption of health checks far easier and standardized across programming languages.

Health Server Architecture

Before diving into the code implementation, let’s briefly walk through the health server flow diagram:

sequenceDiagram
    participant Kubernetes/Service Discovery
    participant Client
    participant Health Server
    participant gRPC Service

    Kubernetes/Service Discovery->>Health Server: Health Check Request
    alt Service is UP
        Health Server->>gRPC Service: Check Internal Health
        gRPC Service-->>Health Server: Status: SERVING
        Health Server-->>Kubernetes/Service Discovery: Status: SERVING
    else Service is DOWN
        Health Server-->>Kubernetes/Service Discovery: Status: NOT_SERVING
    end

    Client->>Health Server: Health Check Request
    Health Server-->>Client: Status: SERVING

sequenceDiagram
    participant Kubernetes/Service Discovery
    participant Client
    participant Health Server
    participant gRPC Service

    Kubernetes/Service Discovery->>Health Server: Health Check Request
    alt Service is UP
        Health Server->>gRPC Service: Check Internal Health
        gRPC Service-->>Health Server: Status: SERVING
        Health Server-->>Kubernetes/Service Discovery: Status: SERVING
    else Service is DOWN
        Health Server-->>Kubernetes/Service Discovery: Status: NOT_SERVING
    end

    Client->>Health Server: Health Check Request
    Health Server-->>Client: Status: SERVING

The diagram above shows how the Health Server acts as a bridge between external systems (Kubernetes/monitoring/etc.) and the gRPC service itself.

Implementation: Practical Steps

Let’s simulate implementing a Health Server on a Go gRPC server (which is also fully supported by Python, Java, Node, and so on, though the syntax may differ).

1. Installing the Packages

1go get google.golang.org/grpc/health
2go get google.golang.org/grpc/health/grpc_health_v1

2. Embedding the Health Server into gRPC

The health protocol has already been created by Google. You simply need to embed the health server into your service.

Code Example:

 1package main
 2
 3import (
 4    "context"
 5    "log"
 6    "net"
 7
 8    "google.golang.org/grpc"
 9    "google.golang.org/grpc/health"
10    healthpb "google.golang.org/grpc/health/grpc_health_v1"
11)
12
13func main() {
14    lis, err := net.Listen("tcp", ":50051")
15    if err != nil {
16        log.Fatalf("failed to listen: %v", err)
17    }
18    s := grpc.NewServer()
19
20    // Initialize the health server
21    healthServer := health.NewServer()
22
23    // Set the health status
24    healthServer.SetServingStatus("", healthpb.HealthCheckResponse_SERVING) // "" means all services
25    // Or set it per-service:
26    // healthServer.SetServingStatus("MyService", healthpb.HealthCheckResponse_SERVING)
27
28    // Register the health server with the gRPC instance
29    healthpb.RegisterHealthServer(s, healthServer)
30
31    log.Println("gRPC server and health server running on :50051")
32    if err := s.Serve(lis); err != nil {
33        log.Fatalf("failed to serve: %v", err)
34    }
35}

Code Explanation:

1. Instantiation: The health server object is initialized using health.NewServer().
2. Set Status: We set the status to SERVING so the health probe receives a signal that the service is ready.
3. Registration: The health server is registered with the gRPC instance running on port 50051.

3. Client Simulation: Performing a Health Check

To check health, you can use grpc-health-probe (a compiled binary/open source) or build your own client.

Example Health Check Client (Go):

 1package main
 2
 3import (
 4    "context"
 5    "fmt"
 6    "time"
 7
 8    "google.golang.org/grpc"
 9    healthpb "google.golang.org/grpc/health/grpc_health_v1"
10)
11
12func main() {
13    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
14    if err != nil {
15        panic(err)
16    }
17    defer conn.Close()
18
19    hc := healthpb.NewHealthClient(conn)
20    resp, err := hc.Check(context.Background(), &healthpb.HealthCheckRequest{})
21    if err != nil {
22        panic(err)
23    }
24    fmt.Printf("Health status: %s\n", resp.Status.String())
25}

Output:

1Health status: SERVING

Health Check Status Table

Advanced: Dynamic HealthStatus for Complex Services

For services that have many dependencies (e.g., a DB, cache, or other services), the health status must be dynamic to reflect the health of those dependencies.

Example of Dynamic Status Updates

 1go func() {
 2    for {
 3        // Assume there's a checkDB() function that returns a bool
 4        dbHealthy := checkDB()
 5        if dbHealthy {
 6            healthServer.SetServingStatus("MyService", healthpb.HealthCheckResponse_SERVING)
 7        } else {
 8            healthServer.SetServingStatus("MyService", healthpb.HealthCheckResponse_NOT_SERVING)
 9        }
10        time.Sleep(10 * time.Second)
11    }
12}()

With this approach, the health server will adjust its “serving” status if a key dependency goes down.

Integration with Kubernetes

Kubernetes can consume the gRPC health endpoint directly using the grpc-health-probe sidecar, which makes things very convenient:

1livenessProbe:
2  exec:
3    command: [
4      "/bin/grpc_health_probe",
5      "-addr=:50051"
6    ]
7  initialDelaySeconds: 5
8  periodSeconds: 10

Best Practices for Implementing a gRPC Health Server

1. Automate the Status: Register a handler that automatically changes the health status according to the state of your dependencies.
2. Multi-Service Health: For services with multiple “gRPC service names,” use per-service status.
3. Integrate with Monitoring: Add custom metrics (Prometheus) to your health checks so they can be monitored comprehensively.
4. Minimize Overhead: Don’t perform heavy (deep-check) inspections on every health probe. Use a cache or a shallow check for readiness probes.

Conclusion

Having a health server in gRPC not only helps orchestrators like Kubernetes perform health checks, but also provides standardization across languages and tools within the cloud native ecosystem. And the implementation is simple, basically plug and play!

By adding a health server, deploying gRPC-based applications to production becomes far more reliable and maintainable!

Have you added a health server to your production gRPC services yet? Share your experience in the comments!

References:

• https://github.com/grpc/grpc/blob/master/doc/health-checking.md
• https://github.com/grpc-ecosystem/grpc-health-probe
• https://blog.grpc.io/2020/05/14/client-side-health-checking.html