57. Case Study: Securing Internal gRPC APIs
Case Study: Securing Internal gRPC APIs
In modern software development, gRPC APIs have grown increasingly popular thanks to their performance, efficient data serialization, and multi-language support. gRPC is widely used for service-to-service communication in microservices architectures, both in cloud-native and enterprise applications. Yet securing gRPC APIs, especially those intended for internal consumption, is often underestimated. Many development teams assume that internal APIs are safe from external threats simply because they live inside a trusted network. In reality, there are plenty of scenarios that can expose internal APIs to serious risk.
This article walks through a case study on applying security to internal gRPC APIs. We’ll explore the main threats, the strategies to mitigate them, and real-world code and architecture examples you can adopt right away to secure your gRPC ecosystem.
Threats to Internal APIs
Many major security incidents over the past two decades have stemmed from placing too much trust in the internal perimeter. Here are the common threats facing internal APIs:
| Threat | Explanation |
|---|---|
| Lateral Movement | An attacker who breaks into one service/VM can pivot to other services. |
| Data Leakage | Without encryption, sensitive data can be sniffed on the internal network. |
| Unauthorized Access | With no authentication/authorization, any service can access an internal API. |
| Privilege Escalation | A service with overly broad permissions can become an entry point for an attacker. |
| Service Impersonation | An attacker poses as an internal service using a custom client. |
Case Study: Payroll Service
Let’s assume a simple microservices architecture inside a fintech company. One critical service is PayrollService, responsible for processing employee salaries. This service should only be accessible by HRService and FinanceService.
Architecture Diagram
flowchart LR
subgraph Internal Network
HRService
FinanceService
PayrollService
end
HRService -- gRPC call --> PayrollService
FinanceService -- gRPC call --> PayrollService
1. Enabling TLS on gRPC
By default, gRPC runs over HTTP/2 without requiring TLS (plaintext), but that leaves it vulnerable to sniffing on the internal LAN. Enabling TLS on all gRPC communication is mandatory, even internally.
TLS Implementation Example in Go
Generate a certificate:
1openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=payroll.internal"PayrollServer.go
1import (
2 "google.golang.org/grpc"
3 "google.golang.org/grpc/credentials"
4)
5
6func main() {
7 creds, err := credentials.NewServerTLSFromFile("cert.pem", "key.pem")
8 if err != nil { log.Fatalf("failed to load cert: %v", err) }
9
10 grpcServer := grpc.NewServer(grpc.Creds(creds))
11 // register your service implementation...
12}Client with TLS:
1conn, err := grpc.Dial(
2 "payroll.internal:50051",
3 grpc.WithTransportCredentials(credentials.NewClientTLSFromFile("cert.pem", "")),
4)Never deploy a gRPC service, not even internally, without TLS!
2. Service Authentication and Mutual TLS (mTLS)
TLS protects data in-transit, but it doesn’t confirm who the client or server actually is. With mutual TLS, both parties must present a valid digital certificate issued by a trusted CA.
Implementing mTLS: PayrollService only accepts requests from whitelisted services.
Generate the CA, server, and client certificates
1# Generate CA
2openssl genrsa -out ca.key 4096
3openssl req -x509 -new -key ca.key -sha256 -out ca.crt -subj "/CN=companyCA"
4
5# Server key and CSR
6openssl genrsa -out payroll.key 4096
7openssl req -new -key payroll.key -out payroll.csr -subj "/CN=payroll.internal"
8openssl x509 -req -in payroll.csr -CA ca.crt -CAkey ca.key -out payroll.crt -CAcreateserial
9
10# Client (for example, HRService)
11openssl genrsa -out hr.key 4096
12openssl req -new -key hr.key -out hr.csr -subj "/CN=hr.internal"
13openssl x509 -req -in hr.csr -CA ca.crt -CAkey ca.key -out hr.crt -CAcreateserialServer (Go):
1creds, _ := credentials.NewTLS(&tls.Config{
2 ClientAuth: tls.RequireAndVerifyClientCert,
3 Certificates: []tls.Certificate{serverCert},
4 ClientCAs: caCertPool, // contains ca.crt
5})
6grpcServer := grpc.NewServer(grpc.Creds(creds))Client (Go):
1creds, _ := credentials.NewTLS(&tls.Config{
2 Certificates: []tls.Certificate{clientCert}, // hr.crt & hr.key
3 RootCAs: caCertPool, // ca.crt
4})
5conn, _ := grpc.Dial(serverAddr, grpc.WithTransportCredentials(creds))With this setup, the payroll service is confident that whoever connects is a trusted client. If any other service tries to connect without a CA-signed certificate, the request is automatically rejected.
3. Authorization Layer: Restricting Which Services May Access
Even though services are already authenticated via mTLS, you still need authorization so that each service only accesses the resources it’s entitled to.
We can apply interceptors on the server to inspect the CN (Common Name) from the client’s certificate.
Interceptor Example in Go:
1import (
2 "google.golang.org/grpc"
3 "google.golang.org/grpc/peer"
4)
5
6func authorizeInterceptor(
7 ctx context.Context,
8 req interface{},
9 info *grpc.UnaryServerInfo,
10 handler grpc.UnaryHandler,
11) (interface{}, error) {
12 p, ok := peer.FromContext(ctx)
13 if !ok { return nil, status.Errorf(codes.PermissionDenied, "No peer found") }
14
15 tlsInfo, ok := p.AuthInfo.(credentials.TLSInfo)
16 if !ok { return nil, status.Errorf(codes.PermissionDenied, "No TLS auth info") }
17
18 cn := tlsInfo.State.PeerCertificates[0].Subject.CommonName
19 if cn != "hr.internal" && cn != "finance.internal" {
20 return nil, status.Errorf(codes.PermissionDenied, "Unauthorized service: %s", cn)
21 }
22
23 return handler(ctx, req)
24}
25
26grpcServer := grpc.NewServer(grpc.Creds(creds), grpc.UnaryInterceptor(authorizeInterceptor))4. Auditing and Rate Limiting
Combining audit logs with rate limiting reduces the room for abuse, especially on a highly sensitive payroll API.
- Implement a server-side rate limiter, for example using the ratelimit library.
- Store an audit log of every request along with the client CN so it’s traceable if an incident occurs.
5. Best Practices Summary
| Practice | Recommendation |
|---|---|
| Transport security | Always use TLS, even on a “trusted” network |
| Service identity | Apply mTLS and an internal PKI |
| Authorization | Validate the CN on every request, whitelist specific services |
| Rate Limit | Throttle the call frequency to sensitive APIs |
| Audit Log | Record every call along with its identity in a protected log |
| Cert management | Automate certificate rotation and distribution, e.g. via cert-manager |
PayrollService Data Protection Flow
sequenceDiagram
participant HR as HRService (Client)
participant PS as PayrollService (Server)
Note left of HR: Hold mTLS Cert (CN=hr.internal)
Note right of PS: Hold mTLS Cert (CN=payroll.internal)
HR->>PS: Initiate gRPC TLS handshake
HR-->>PS: Present client cert
PS-->>HR: Present server cert
PS->>PS: Verify client CN == Whitelist
HR->>PS: PayrollRequest (succeeds if mTLS & CN OK)
PS-->>HR: PayrollResponse
Note over HR,PS: All data encrypted & authorized end-to-end
Conclusion
Even though gRPC is typically deployed for internal services, assuming that an internal API is safe is like assuming a house needs no doors simply because it sits in a “safe” neighborhood. The reality is that cyber threats keep evolving; opportunities for lateral movement, sniffing, and even data exfiltration can arise at any time. Applying TLS, mTLS, per-service authorization, and audit logging is the minimum viable security for an internal gRPC ecosystem.
Start by enabling TLS today. Then move on to mTLS, and after that, automate certificate deployment through proper secret management. Cultivate a zero trust mindset, staying alert to attacks from both inside and outside the perimeter. When an incident happens, the audit log is your lifeline. Hopefully this PayrollService case study leaves you better prepared to build internal gRPC services that are secure by design.
References: