69. Case Study: Retry and Backoff Strategy
69. Case Study: Retry and Backoff Strategy
As a backend engineer, or software engineers in general, we can’t escape interacting with external services: payment gateway APIs, databases, queue services, and third-party APIs that are beyond our control. One of the challenges we often run into is the flaky service, that is, a remote service that sometimes fails to respond, runs slowly, or fails completely for a short period. This is exactly where retry and backoff strategies become so important.
This article walks through the concepts of retry and backoff with a case study, complete with code examples, simulations, as well as flow and performance analysis.
The Problem: A Flaky Remote Service
Suppose we have a simple service: an e-wallet application that needs to validate a voucher against a Third-Party Voucher Service every time a user redeems one.
However, the Voucher Service frequently times out or returns a 500 Internal Server Error at random.
If our system simply fails outright whenever the voucher service errors out, the user experience is obviously poor. But if we retry naively without any strategy, it can be even worse: the load spikes and the service goes down because of the thundering herd problem.
Simple Retry: Just One More Time
First, let’s start with the simplest strategy: retrying a failed request at most once.
Code Example (Pseudo-Python)
1import requests
2
3def validate_voucher(voucher_code):
4 url = f"https://voucher.example.com/voucher/{voucher_code}/validate"
5 for attempt in range(2): # Maximum of 2 attempts (1 retry)
6 try:
7 response = requests.get(url, timeout=2) # 2 second timeout
8 response.raise_for_status()
9 return response.json()['valid']
10 except (requests.HTTPError, requests.Timeout) as e:
11 print(f"Attempt {attempt+1} failed: {str(e)}")
12 if attempt == 1:
13 raise
14 # Retry immediately within the loopThe code above works, but it raises a number of important questions:
- Would it be better to perform more retries?
- How quickly should we retry? Immediately, or wait a moment first?
- What happens if every instance in the cluster retries at the same time?
Excessive Retrying Can Be Dangerous
Imagine we perform 5 retries on the spot the moment something fails. If the root cause of the error is remote service overload, mass retries will only make the situation worse. This is known as a retry storm or thundering herd.
This naive solution can be illustrated with the following diagram:
sequenceDiagram
participant User
participant Service
participant VoucherService
User->>Service: Redeem Voucher
Service->>VoucherService: Validate Voucher
VoucherService--xService: 500 Error/Timeout
Service->>VoucherService: Retry (immediately)
VoucherService--xService: 500 Again
...repeat...
Introducing the Backoff Strategy
Backoff means waiting a moment before performing a retry. There are several backoff patterns you can use:
- Fixed Wait: Always wait X seconds for every retry.
- Incremental Wait: With each retry, the wait time increases.
- Exponential Backoff: The wait time increases exponentially (1s, 2s, 4s, 8s, …).
- Exponential + Random Jitter: Adds randomness to avoid all clients retrying simultaneously.
Let’s implement exponential backoff with jitter.
Case Study With Exponential Backoff & Jitter
Code Example (Python)
1import requests
2import random
3import time
4
5def validate_voucher_with_backoff(voucher_code, max_retries=5, base_delay=0.5):
6 url = f"https://voucher.example.com/voucher/{voucher_code}/validate"
7 for attempt in range(max_retries):
8 try:
9 response = requests.get(url, timeout=2)
10 response.raise_for_status()
11 return response.json()['valid']
12 except (requests.HTTPError, requests.Timeout) as e:
13 print(f"Attempt {attempt+1} failed: {str(e)}")
14 # Exponential Backoff: delay = base_delay * 2 ^ attempt
15 delay = base_delay * (2 ** attempt)
16 # Add random jitter +/- 0..base_delay
17 jitter = random.uniform(0, base_delay)
18 total_delay = delay + jitter
19 print(f"Retrying in {total_delay:.2f} seconds...")
20 time.sleep(total_delay)
21 raise Exception("All attempts failed")Diagram Visualization
gantt
title Retry dan Backoff Timeline
dateFormat X
section Retry Attempts
Attempt#1 :done, 0, 1
Backoff#1 :active, 1, 0.6
Attempt#2 :done, 1.6, 1
Backoff#2 :active, 2.6, 1.2
Attempt#3 :done, 3.8, 1
Simulation: The Effect of Backoff and Jitter
Imagine 1000 users redeeming a voucher all at once.
- Without backoff: Every retry happens within <1 second.
- With exponential backoff + jitter: Retries are spread out over several seconds, reducing the peak load on the Voucher Service.
| Scenario | Peak Load on Remote Service | Average Time to Success |
|---|---|---|
| No Retry | Low | Low |
| Retry Immediate (No Backoff) | Very high, risk of a storm | Faster (if the service is healthy) |
| Exponential Backoff + Jitter | Lower, load is spread out | Slower, but more robust |
How Do You Choose the Right Retry and Backoff?
There’s no single definitive answer. Here are a few practical principles:
- Identify which errors are safe to retry. Retry only for temporary failures: timeouts, 5xx, connection resets. Errors such as authentication and validation should fail fast.
- Limit the number of retries (usually 3-5 times).
- Backoff: use exponential (or at least incremental) backoff, never fixed-delay alone.
- Jitter: randomize the wait time so that retries don’t happen all at once.
- Early termination logic: if the deadline is already near, cancel all (use a context with a timeout in Go, or a deadline in Java).
Implementation Example in Go
1package main
2
3import (
4 "fmt"
5 "math/rand"
6 "net/http"
7 "time"
8)
9
10func ValidateVoucherWithBackoff(voucherCode string, maxRetries int, baseDelay time.Duration) (bool, error) {
11 url := fmt.Sprintf("https://voucher.example.com/voucher/%s/validate", voucherCode)
12 var client = &http.Client{Timeout: 2 * time.Second}
13
14 for attempt := 0; attempt < maxRetries; attempt++ {
15 resp, err := client.Get(url)
16 if err == nil && resp.StatusCode == http.StatusOK {
17 return true, nil // Simulated return
18 }
19 delay := baseDelay * (1 << attempt) // Exponential
20 jitter := time.Duration(rand.Int63n(int64(baseDelay)))
21 totalDelay := delay + jitter
22 fmt.Printf("Attempt %d failed. Backoff %s\n", attempt+1, totalDelay)
23 time.Sleep(totalDelay)
24 }
25 return false, fmt.Errorf("all attempts failed")
26}Takeaways
- Retrying without a strategy is just as bad as having no retry at all.
- Exponential Backoff + Jitter is an industry best practice pattern for mitigating problems with unstable remote services.
- Analyze the scenarios realistically: latency goes up, but the system becomes more stable.
- Instrumentation (monitoring and logging) is a must so that you understand the root cause—too many retries can indicate a systemic problem or a struggling dependency.
Conclusion
Retry and backoff are simple but crucial patterns in distributed systems. Understanding when and how to retry isn’t just a matter of coding; it’s also about designing resilience and reliability into our systems.
Hopefully the case study and code examples above will serve you well when you’re dealing with an unreliable upstream service—because in production, “Trust (and retry) but verify!” is a powerful technique for surviving the microservices world that is never fully reliable.
Feel free to discuss and share the retry strategies you’ve put into practice—battle-tested experience is what makes us tougher engineers.
// Cheers!