82. Testing with bufconn Without a Network

When building gRPC-based applications, one of the biggest challenges is testing reliably without having to deal with the complexity of a real network. Typically, gRPC tests are performed by simulating a server running on a specific port, then having the client send requests to it. However, this approach comes with several limitations, including:

• It’s slow, because you actually have to open a network socket
• It’s prone to port race conditions, especially when tests run in parallel
• It can be disrupted by network firewalls, resource limits, and so on

In the Go world, there’s an elegant solution to this problem: bufconn , short for buffered connection. With bufconn, we can have a gRPC client and server communicate through a memory buffer (similar to a pipe), without actually opening a network connection. This article covers the concept, the practice, the pros and cons, and a walkthrough of using bufconn for gRPC testing in Go.

What Is bufconn?

In simple terms, bufconn is a package that provides an implementation of net.Listener, but its transport uses a memory buffer instead of the network. This enables dependency injection during testing simply by swapping the server’s listener to bufconn.Listen instead of net.Listen.

The illustration looks roughly like this:

flowchart LR
    A[gRPC Client] --(memory buffer)--> B[gRPC Server]
    subgraph Normal Network
        style A fill:#f9f,stroke:#333,stroke-width:2px
        style B fill:#bbf,stroke:#333,stroke-width:2px
    end
    A2[gRPC Client] --TCP socket--> B2[gRPC Server]
    subgraph Bufconn
        style A2 fill:#f6f,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
        style B2 fill:#bdf,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
    end

flowchart LR
    A[gRPC Client] --(memory buffer)--> B[gRPC Server]
    subgraph Normal Network
        style A fill:#f9f,stroke:#333,stroke-width:2px
        style B fill:#bbf,stroke:#333,stroke-width:2px
    end
    A2[gRPC Client] --TCP socket--> B2[gRPC Server]
    subgraph Bufconn
        style A2 fill:#f6f,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
        style B2 fill:#bdf,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
    end

In the diagram above, the left side shows testing via a memory buffer (bufconn), while the right side uses a real network (TCP socket).

Why Use bufconn?

There are several reasons why you should start considering bufconn in your testing workflow:

Example Scenario: Testing a gRPC Calculator Service

As an illustration, suppose we have a simple gRPC service called Calculator with a single Add method.

1. Service Definition (proto)

 1// calculator.proto
 2syntax = "proto3";
 3package calculator;
 4
 5service Calculator {
 6  rpc Add (AddRequest) returns (AddReply);
 7}
 8
 9message AddRequest {
10  int32 a = 1;
11  int32 b = 2;
12}
13
14message AddReply {
15  int32 result = 1;
16}

Generate the Go code (make sure you’ve already installed protoc-gen-go-grpc and protoc-gen-go):

1protoc --go_out=. --go-grpc_out=. calculator.proto

2. Go Server Implementation

 1// calculator_server.go
 2package main
 3
 4import (
 5  pb "path/to/calculatorpb"
 6  "context"
 7)
 8
 9type calculatorServer struct {
10  pb.UnimplementedCalculatorServer
11}
12
13func (s *calculatorServer) Add(ctx context.Context, req *pb.AddRequest) (*pb.AddReply, error) {
14  sum := req.A + req.B
15  return &pb.AddReply{Result: sum}, nil
16}

3. Setting Up Testing with Bufconn

Let’s write an integration test without a network:

 1// calculator_test.go
 2package main
 3
 4import (
 5  "context"
 6  "log"
 7  "net"
 8  "testing"
 9  "google.golang.org/grpc"
10  "google.golang.org/grpc/test/bufconn"
11
12  pb "path/to/calculatorpb"
13)
14
15const bufSize = 1024 * 1024
16
17var lis *bufconn.Listener
18
19func bufDialer(context.Context, string) (net.Conn, error) {
20  return lis.Dial()
21}
22
23func startTestGrpcServer() {
24  lis = bufconn.Listen(bufSize)
25  s := grpc.NewServer()
26  pb.RegisterCalculatorServer(s, &calculatorServer{})
27  go func() {
28    if err := s.Serve(lis); err != nil {
29      log.Fatalf("Server exited with: %v", err)
30    }
31  }()
32}
33
34func TestAdd_WithBufconn(t *testing.T) {
35  startTestGrpcServer()
36
37  ctx := context.Background()
38  conn, err := grpc.DialContext(
39    ctx,
40    "bufnet",
41    grpc.WithContextDialer(bufDialer),
42    grpc.WithInsecure(),
43  )
44  if err != nil {
45    t.Fatalf("Failed to dial bufnet: %v", err)
46  }
47  defer conn.Close()
48
49  client := pb.NewCalculatorClient(conn)
50  resp, err := client.Add(ctx, &pb.AddRequest{A: 10, B: 30})
51  if err != nil {
52    t.Fatalf("Add failed: %v", err)
53  }
54  if resp.Result != 40 {
55    t.Fatalf("Expected 40, got %d", resp.Result)
56  }
57}

4. Simulating Test Execution

Let’s look at the internal flow when the test runs:

sequenceDiagram
    participant T as Test Runner (Go `testing`)
    participant C as gRPC Client
    participant S as gRPC Server (in memory buffer)

    T->>S: Start the gRPC server on bufconn.Listener
    T->>C: Create a gRPC client connection via bufDialer
    C->>S: Send the Add(10, 30) request (via memory buffer)
    S->>S: Process and respond with AddReply(40)
    S->>C: Send the reply back to the client (via buffer)
    C->>T: Result is returned to the test runner

sequenceDiagram
    participant T as Test Runner (Go `testing`)
    participant C as gRPC Client
    participant S as gRPC Server (in memory buffer)

    T->>S: Start the gRPC server on bufconn.Listener
    T->>C: Create a gRPC client connection via bufDialer
    C->>S: Send the Add(10, 30) request (via memory buffer)
    S->>S: Process and respond with AddReply(40)
    S->>C: Send the reply back to the client (via buffer)
    C->>T: Result is returned to the test runner

The biggest advantage: The test runs entirely in memory, fast, with no need for a port and free from race conditions or OS interference.

Tips for Production Use

• Use bufconn only for unit/integration tests, not real production!
• It’s a great fit for testing services with a lot of network dependencies or that are hard to mock.
• For tests with real interactions (TLS, firewalls), you still need an e2e test over a real socket.

Conclusion

Testing with bufconn opens up a new way to speed up and tidy up the gRPC testing ecosystem. We no longer have to fret over port clashes or race conditions in integration tests. Just inject a bufconn.Listener, and all of your tests run in memory without a single byte being sent over the real network.

A summary of the benefits of using bufconn for gRPC testing:

• Eliminates network dependencies
• Improves test determinism and speed
• Safe for parallel tests and CI/CD pipelines

Remember: Use bufconn for integration/unit-level tests, but don’t forget e2e tests in a real environment for full coverage.

Happy experimenting with bufconn in your gRPC testing workflow! 🚀

Further Reading:

• gRPC bufconn
• Official gRPC-Go testing