Controlling Data Flow with Streams in Golang

In an era of real-time, large-scale systems, managing data flow has become an essential requirement in nearly every application—from backends that handle requests from thousands of users, to ETL processes over massive datasets, to IoT applications whose data keeps flowing in continuously. One of the fundamental concepts that helps developers tackle this problem is the stream.

This article will explore how to control data flow using streams in Go: starting from why streams matter, how they work in principle, all the way to implementing techniques such as buffering, throttling, and backpressure.

Why Use Streams?

Without streams, applications tend to read and process data in bulk. This leads to several problems:

• High memory consumption: Reading a large file all at once can cause OOM.
• Slow response: Waiting for all the data to finish processing before moving on.
• Poor scalability: The bulk approach is difficult to scale.

With streams, we only fetch, process, and send data piece by piece, which gives us:

• Memory efficiency
• Faster responsiveness
• Natural scalability

The Anatomy of a Stream

In general, a stream consists of several main components:

Flow Diagram

flowchart LR
    A[Source] --> B[Buffer]
    B --> C[Operator]
    C --> D[Sink / Consumer]

flowchart LR
    A[Source] --> B[Buffer]
    B --> C[Operator]
    C --> D[Sink / Consumer]

Case Study: Reading a Large File with Streams

 1package main
 2
 3import (
 4    "bufio"
 5    "fmt"
 6    "os"
 7)
 8
 9func main() {
10    file, err := os.Open("bigdata.txt")
11    if err != nil {
12        panic(err)
13    }
14    defer file.Close()
15
16    scanner := bufio.NewScanner(file)
17    for scanner.Scan() {
18        line := scanner.Text()
19        fmt.Println("Line:", line)
20    }
21
22    if err := scanner.Err(); err != nil {
23        fmt.Println("Error:", err)
24    }
25}

• The file is read line by line.
• Suitable for large files because only one line is loaded into memory at a time.

Buffering and Throttling

Buffering

Use bufio.Reader for manual buffering:

1reader := bufio.NewReader(file)
2line, _ := reader.ReadString('\n')

Throttling

Use time.Sleep to control the consumption rate:

1for scanner.Scan() {
2    fmt.Println(scanner.Text())
3    time.Sleep(100 * time.Millisecond) // simulate throttling
4}

Backpressure in Go

Go does not have a built-in backpressure mechanism on channels, but you can control it through buffer capacity and blocking behavior.

 1func producer(ch chan<- int) {
 2    for i := 0; i < 10; i++ {
 3        fmt.Println("Producing", i)
 4        ch <- i
 5    }
 6    close(ch)
 7}
 8
 9func consumer(ch <-chan int) {
10    for data := range ch {
11        fmt.Println("Consuming", data)
12        time.Sleep(200 * time.Millisecond) // simulate slow processing
13    }
14}
15
16func main() {
17    ch := make(chan int, 2)
18    go producer(ch)
19    consumer(ch)
20}

With a small buffer and slow processing, Go automatically delays the producer from sending more data (natural backpressure).

Bulk vs. Stream Comparison

Visual Simulation of Backpressure

flowchart RL
    P(Producer) -->|Data| CH[Channel Buffer]
    CH -->|Jika tidak penuh| C(Consumer)
    C -->|Lambat memproses| Delay
    CH -->|Jika penuh| P[Producer Stalled]

flowchart RL
    P(Producer) -->|Data| CH[Channel Buffer]
    CH -->|Jika tidak penuh| C(Consumer)
    C -->|Lambat memproses| Delay
    CH -->|Jika penuh| P[Producer Stalled]

Conclusion

Streams are an essential tool in building modern applications that are efficient and scalable. By breaking data into small pieces, we can control memory load, improve system responsiveness, and avoid bottlenecks.

Go provides tools such as channels, goroutines, bufio, and blocking operations that are quite effective for building stream-based systems, especially for I/O and data processing pipelines.

If you’d like to continue into applying streams in the context of HTTP/gRPC, Kafka, or log collectors, just let me know. We can keep the article going!