76. Connecting a gRPC Service with Kafka

When building modern distributed systems, the communication patterns between microservices become extremely crucial. Two popular technologies that are often used are gRPC and Apache Kafka. Combining these two technologies offers speed, stability, and scalability for enterprise systems. In this article, I’ll discuss how to connect a gRPC service with Kafka—from the architecture and code implementation all the way to a testing simulation.

Why Do We Need to Integrate gRPC with Kafka?

gRPC is a modern Remote Procedure Call (RPC) communication system based on HTTP/2 and Protocol Buffers, which is efficient for synchronous communication between services. Meanwhile, Kafka is a powerful message broker for asynchronous message delivery at massive scale.

The typical pattern that is often used:

• gRPC as the API service layer (a front-facing service that performs validation and synchronous processing).
• Kafka as the event bus for asynchronous communication and real-time/batch data processing by downstream services.

This way, every request received by gRPC can be pushed as an event to Kafka—opening the door for other services to react to that event in a loosely coupled manner.

Case Study: Order Service

Imagine we are building an Order Service (gRPC) that, every time a user creates a new order, publishes an order event to Kafka. Other systems such as the Inventory Service and Notification Service will consume this event for further processing.

High-Level Architecture

Let’s look at the system flow using a Mermaid diagram:

sequenceDiagram
    participant Client
    participant gRPC Service
    participant Kafka
    participant Consumer Service

    Client->>gRPC Service: gRPC call (CreateOrder)
    gRPC Service->>Kafka: Publish OrderCreated event
    Kafka->>Consumer Service: Consumer receives OrderCreated event

sequenceDiagram
    participant Client
    participant gRPC Service
    participant Kafka
    participant Consumer Service

    Client->>gRPC Service: gRPC call (CreateOrder)
    gRPC Service->>Kafka: Publish OrderCreated event
    Kafka->>Consumer Service: Consumer receives OrderCreated event

In brief:

1. The client sends a CreateOrder request to the gRPC service.
2. The gRPC service validates it, then pushes an OrderCreated event to a Kafka topic.
3. Other services act as consumers of that topic.

Environment Setup

For this simulation, we’ll use the following stack:

• Go as the programming language (because the gRPC and Kafka bindings in Go are very solid)
• protoc to generate stubs from the proto file
• sarama as the Kafka client in Go

The code below is also relevant for stacks other than Go, since the principle is the same (publish an event to Kafka after handling the gRPC call).

1. Protobuf Definition

File: order.proto

 1syntax = "proto3";
 2
 3package order;
 4
 5service OrderService {
 6  rpc CreateOrder (OrderRequest) returns (OrderResponse);
 7}
 8
 9message OrderRequest {
10  string item = 1;
11  int32 qty = 2;
12  string user_id = 3;
13}
14
15message OrderResponse {
16  string order_id = 1;
17  string status = 2;
18}

Generate the Go code for that service and those messages.

2. Implementing the gRPC Server with a Kafka Publisher

File: main.go

 1package main
 2
 3import (
 4	"context"
 5	"log"
 6	"net"
 7
 8	pb "github.com/yourorg/orderpb"
 9	"github.com/Shopify/sarama"
10	"google.golang.org/grpc"
11)
12
13type server struct {
14	producer sarama.SyncProducer
15	pb.UnimplementedOrderServiceServer
16}
17
18func (s *server) CreateOrder(ctx context.Context, req *pb.OrderRequest) (*pb.OrderResponse, error) {
19	orderID := generateOrderID() // random function, e.g. UUID
20
21	event := map[string]interface{}{
22		"order_id": orderID,
23		"item":     req.Item,
24		"qty":      req.Qty,
25		"user_id":  req.UserId,
26	}
27
28	// Serialize to JSON
29	marshalled, err := json.Marshal(event)
30	if err != nil {
31		return nil, err
32	}
33
34	// Publish to the Kafka topic
35	message := &sarama.ProducerMessage{
36		Topic: "order_events",
37		Value: sarama.ByteEncoder(marshalled),
38	}
39
40	_, _, err = s.producer.SendMessage(message)
41	if err != nil {
42		return nil, err
43	}
44
45	return &pb.OrderResponse{
46		OrderId: orderID,
47		Status:  "CREATED",
48	}, nil
49}
50
51func main() {
52	// Initialize the Kafka producer
53	config := sarama.NewConfig()
54	config.Producer.Return.Successes = true
55	producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config)
56	if err != nil {
57		log.Fatalf("Failed to start Sarama producer: %v", err)
58	}
59
60	lis, err := net.Listen("tcp", ":50051")
61	if err != nil {
62		log.Fatalf("Failed to listen: %v", err)
63	}
64	grpcServer := grpc.NewServer()
65	pb.RegisterOrderServiceServer(grpcServer, &server{producer: producer})
66
67	log.Println("gRPC & Kafka server running at :50051 ...")
68	if err := grpcServer.Serve(lis); err != nil {
69		log.Fatalf("Failed to serve: %v", err)
70	}
71}

Code Explanation

• Every time an order is created successfully, the event is pushed to the Kafka topic.
• gRPC still returns a synchronous response to the client (“CREATED”).
• The code is easy to extend to also handle database synchronization, logging, and so on.

3. Kafka Consumer (Simulation)

This order event from Kafka can be consumed by another service like the following:

 1consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, nil)
 2if err != nil {
 3    panic(err)
 4}
 5partitionConsumer, err := consumer.ConsumePartition("order_events", 0, sarama.OffsetNewest)
 6if err != nil {
 7    panic(err)
 8}
 9for message := range partitionConsumer.Messages() {
10    var event map[string]interface{}
11    if err := json.Unmarshal(message.Value, &event); err == nil {
12        log.Printf("Order Event received: %+v\n", event)
13        // Do something else: update stock, send email, etc.
14    }
15}

Testing Simulation

Let’s create a testing simulation using a table of steps along with sample log output.

Example log output:

12024/06/12 11:00:01 gRPC & Kafka server running at :50051 ...
22024/06/12 11:00:08 Order Event received: map[item:"Coffee" order_id:"123" qty:2 user_id:"abcde"]

Benefits of This Architecture

Advantages

• Scalability: The order event can be received by many consumers without adding load to the gRPC API.
• Decoupling: Downstream services can subscribe/unsubscribe without changing the OrderService code.
• Reliability & Retries: Kafka supports re-delivery when a consumer fails.
• Asynchronous processing: Heavy processes (e.g. updating stock, sending email) don’t block the client request.

Disadvantages

• Infrastructure Requirements: You need to deploy and manage a Kafka cluster.
• Error Handling: You need monitoring/retry policies in case publishing to Kafka fails.
• Duplicate Events: Downstream services need to be idempotent (an event may be received more than once).

Conclusion

By combining gRPC (for synchronous APIs) and Kafka (for asynchronous events), we can build microservices that are scalable and loosely coupled. The techniques explained above have become a best practice at many large tech companies.

Here is a 📝 checklist for a production-ready gRPC + Kafka integration:

• Use a schema registry for event contracts (to avoid broken consumers)
• Logging & distributed tracing at every step
• Monitoring of event publishing and delivery
• Implement retry/retry policies on both the publisher and the consumer

This integration is suitable for a variety of use cases — from order management and payments to email/SMS notifications.
Give this structure a try in your own microservices project, and enjoy how easy it makes scaling your services 🚀.