87. Case Study: Debugging Streaming Issues

Streaming data in real time has increasingly become a fundamental requirement, whether for video and audio services or IoT sensor data. Yet anyone who has ever built a streaming pipeline has surely run into strange problems: lag, lost data, duplication, and even crashes with no obvious cause. In this article, I’ll share a real-world case study on dealing with issues in a Kafka-based data streaming system, from identification and bug replication all the way to finally pinpointing the root cause and fixing it.

Background

The streaming system we built was designed to transfer data from service A (producer) to service B (consumer) through Apache Kafka. This data was critical: losing even a single event could leave user data inconsistent in downstream services. Here is a simple architecture diagram:

flowchart LR
    A[Service A
Kafka Producer] -- PUSH --> K[Kafka Topic]
    K -- SUBSCRIBE --> B[Service B
Kafka Consumer]

flowchart LR
    A[Service A<br>Kafka Producer] -- PUSH --> K[Kafka Topic]
    K -- SUBSCRIBE --> B[Service B<br>Kafka Consumer]

The Problem That Emerged

Periodically, the team found that the consumer in Service B would sometimes receive duplicate data, or not receive any data at all. At first, we assumed the problem was in Service B itself — network lag, slow consumption, and so on. But it turned out the problem was more complex.

Symptoms Observed:

• Consumer lag in the metrics spiked for no apparent reason.
• Jumps occurred in the offset (the offset leaped from n to n+10).
• Sometimes the producer appeared to publish successfully without any error.

Step 1: Identifying the Problem

The first step: observation and logging. We added extensive logging on both sides:

• Producer: Logging every message that succeeded/failed along with its offset.
• Kafka: Monitoring partition logs, disk usage, and internal metrics such as under-replicated partitions.
• Consumer: Logging every offset commit, incoming message, and connection status.

We also wrote a simple script to simulate a consumer:

 1from kafka import KafkaConsumer
 2import json
 3
 4consumer = KafkaConsumer(
 5    'my-topic',
 6    bootstrap_servers=['kafka:9092'],
 7    group_id='debug-group',
 8    auto_offset_reset='earliest',
 9    enable_auto_commit=True
10)
11
12for msg in consumer:
13    print(f'Offset: {msg.offset}, Key: {msg.key}, Value: {msg.value}')

Step 2: Analyzing the Metrics

We found the following anomaly:

Insight:
At 10:01 the producer offset increased, but the offset in Kafka temporarily failed to rise; this means some messages never made it into Kafka. After a few minutes, the Kafka offset “returned to normal.”

Step 3: Simplifying the Problem

We wrote a script to simulate a burst when sending the following messages:

 1from kafka import KafkaProducer
 2import json
 3import time
 4
 5producer = KafkaProducer(
 6    bootstrap_servers=['kafka:9092'],
 7    value_serializer=lambda v: json.dumps(v).encode('utf-8')
 8)
 9
10for i in range(120):
11    data = {'event': f'event-{i}', 'timestamp': time.time()}
12    producer.send('my-topic', value=data, key=str(i).encode())
13    time.sleep(0.01 if i < 100 else 0.0001)  # Simulate burst
14producer.flush()

The result: under a high burst, messages were dropped without any error, so the application assumed the messages had been sent successfully.

Step 4: Investigating Kafka and the Network

A very powerful Kafka feature like acks=1 makes message delivery “fire-and-forget,” but insecure. Our setup, as it turned out, relied on the default ack, namely acks=1. For critical applications, you should use acks=all to ensure that messages are truly received by all broker replicas.

Finding the Bottleneck:

• We found saturation on the broker’s disk.
• The network was briefly disrupted on a certain reroute.
• Kafka marked some partitions as “under-replicated,” causing a portion of the events to not be durable.

A flow diagram of the Kafka message write process:

sequenceDiagram
    participant P as Producer
    participant B as Broker
    participant R as Replica

    P->>B: Send message (acks=1)
    alt acks=1
        B->>P: Respond OK after leader write
    else acks=all
        B->>R: Replicate message
        R->>B: Ack
        B->>P: Respond OK after all acks
    end

sequenceDiagram
    participant P as Producer
    participant B as Broker
    participant R as Replica

    P->>B: Send message (acks=1)
    alt acks=1
        B->>P: Respond OK after leader write
    else acks=all
        B->>R: Replicate message
        R->>B: Ack
        B->>P: Respond OK after all acks
    end

With acks=1, a message may in fact not yet be durable across all brokers; if the leader fails, the data is lost. This is very dangerous for a system that requires high durability.

Step 5: Root Cause and Fix

We ran several experiments:

1. Changing acks to all in the producer:

   python

   Copy

   1producer = KafkaProducer(
   2    bootstrap_servers=['kafka:9092'],
   3    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
   4    acks='all'
   5)

2. Monitoring Partition Replica Lags actively.

3. Ensuring Disk Health: Disk capacity and IOPS must be maintained so the broker doesn’t get stuck.

4. Active Retry: Add retry logic for non-transient errors.

After the changes above, message loss almost disappeared, lag dropped drastically, and bursty messages no longer caused silent failures.

Further Insight: Error Handling in the Code

Another strategy: mitigation at the application level. For example, if a send fails or times out:

1from kafka.errors import KafkaTimeoutError
2
3try:
4    producer.send('my-topic', value=data)
5except KafkaTimeoutError as e:
6    # Implement retry logic or fallback
7    print('Timeout, retrying...')

And on the consumer side, implement manual offset commits to make it more reliable:

1consumer = KafkaConsumer(
2    'my-topic',
3    bootstrap_servers=['kafka:9092'],
4    group_id='debug-group',
5    enable_auto_commit=False
6)
7for msg in consumer:
8    process(msg)
9    consumer.commit()

Conclusion

Streaming problems are often a combination of distributed architecture, network flakiness, and wrong assumptions about durability. Armed with thorough observation, good logging, and benchmarks that cover edge-case scenarios, we can find the root cause in seemingly “trivial” places such as the ack setting or disk IOPS.

Takeaways for other engineers:

• Always review your Kafka producer config, especially acks and retries.
• Monitoring partitions, disk, and replica health is a must.
• Simulate traffic bursts and failure modes as early as possible.
• Document your investigation so other teams don’t fall into the same trap!

Always remember: streaming isn’t just about “data passing through,” it’s about guaranteeing the consistency and integrity of our users’ data. Debug properly, debug deeply.

Happy debugging!
Leave a comment if you’ve ever run into a similar problem. I’d love to hear your solutions!