29 Case Study: Streaming File Upload
Streaming File Upload with Go and gRPC
Uploading large files is a classic challenge in backend development, especially when the system must be able to handle hundreds of MB or even GB in a single session. When done conventionally (buffer all then store), the risk of memory overload and failure is very high. This is exactly where the streaming upload approach becomes important.
This article will discuss how to implement streaming file upload using Go and gRPC streaming. We’ll simulate an API that uploads a large file from the client to the server, handled in a chunked manner.
Why Choose gRPC Streaming?
With gRPC streaming, we can:
- Send data incrementally in chunks.
- Avoid excessive memory usage.
- Provide upload progress feedback.
- Manage large file uploads efficiently and in a scalable way.
Architecture and Streaming Upload Flow
sequenceDiagram
participant Client
participant gRPCServer
participant Disk
Client->>gRPCServer: stream Upload(chunk)
loop Selama upload
gRPCServer->>Disk: write(chunk)
end
gRPCServer-->>Client: UploadResponse(status)
1. Protobuf Definition
1syntax = "proto3";
2
3package upload;
4
5service FileService {
6 rpc Upload(stream UploadRequest) returns (UploadResponse);
7}
8
9message UploadRequest {
10 bytes chunk = 1;
11 string filename = 2; // only sent in the first chunk
12}
13
14message UploadResponse {
15 string message = 1;
16 bool success = 2;
17}2. Implementing the gRPC Server in Go
a. Server Setup
1package main
2
3import (
4 "context"
5 "fmt"
6 "io"
7 "log"
8 "os"
9 "google.golang.org/grpc"
10 "net"
11 pb "your_project/proto/upload"
12)
13
14type server struct {
15 pb.UnimplementedFileServiceServer
16}
17
18func (s *server) Upload(stream pb.FileService_UploadServer) error {
19 var file *os.File
20 firstChunk := true
21
22 for {
23 req, err := stream.Recv()
24 if err == io.EOF {
25 return stream.SendAndClose(&pb.UploadResponse{
26 Message: "Upload selesai",
27 Success: true,
28 })
29 }
30 if err != nil {
31 return err
32 }
33
34 if firstChunk {
35 file, err = os.Create("uploads/" + req.Filename)
36 if err != nil {
37 return err
38 }
39 defer file.Close()
40 firstChunk = false
41 }
42
43 _, err = file.Write(req.Chunk)
44 if err != nil {
45 return err
46 }
47 }
48}
49
50func main() {
51 lis, err := net.Listen("tcp", ":50051")
52 if err != nil {
53 log.Fatalf("gagal listen: %v", err)
54 }
55
56 s := grpc.NewServer()
57 pb.RegisterFileServiceServer(s, &server{})
58 log.Println("Server gRPC berjalan di port 50051")
59 s.Serve(lis)
60}3. Simulating the Streaming Upload Client
1conn, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
2defer conn.Close()
3
4client := pb.NewFileServiceClient(conn)
5stream, _ := client.Upload(context.Background())
6
7f, _ := os.Open("large.pdf")
8buf := make([]byte, 1024*32) // 32 KB per chunk
9
10stream.Send(&pb.UploadRequest{Filename: "large.pdf"})
11
12for {
13 n, err := f.Read(buf)
14 if err == io.EOF {
15 break
16 }
17 stream.Send(&pb.UploadRequest{Chunk: buf[:n]})
18}
19
20res, _ := stream.CloseAndRecv()
21fmt.Println("Respon server:", res.Message)4. Comparison: Buffer vs Streaming
| Parameter | Buffer Upload | gRPC Streaming |
|---|---|---|
| Memory Usage | High (file stored in memory) | Stable (per chunk) |
| Time to Write | After all data has been received | Immediately during upload |
| Scalability | Poor | Good (supports parallel) |
| Complexity | Low | Medium |
5. Production Tips
- ✅ Validate file size on both the client and the server.
- ✅ Limit the chunk size and maximum file size (use metadata).
- ✅ Add TLS + metadata for authentication.
- ✅ Stream directly to cloud storage (for example, Google Cloud Storage or S3) for efficiency.
- ✅ Logging and tracing are extremely helpful for debugging.
Conclusion
Streaming file upload with gRPC in Go delivers high efficiency and granular control over the data flow. This approach is a great fit for applications that serve large files, handle many users, or require high performance.
gRPC streaming is a powerful tool when used correctly. If your application’s architecture involves uploading large files, it’s time to switch to this approach.
Want me to help you set up a file stream upload module with metadata validation and Prometheus monitoring? Just let me know! 🚀