AI Incident Playbook Golang: Detect, Triage, Rollback AI Failures in Production
A complete guide to handling AI feature incidents in Golang production. Automatic detection, triage, rollback, and post-mortem for reliable AI-powered Go services.
AI Incident Playbook: When AI Features Fail in Production
An AI incident playbook is a mandatory need because AI features fail in ways that differ from ordinary software. A bug in Go code produces a deterministic error — easy to detect, easy to roll back. AI failures can be far more subtle: output that looks valid but is semantically wrong, quality that degrades slowly, or costs that explode without warning.
In this article we build a comprehensive incident playbook for AI-powered Go services — from classifying failure types, an automatic detection system, severity classification, a triage decision tree, fallback and circuit breaker strategies, all the way to a post-mortem process that turns every incident into material for improvement.
18.01 The Different Types of AI Failure
Handling AI failures starts with recognizing that they aren’t uniform. The classification below splits AI failures into four categories based on how easy they are to detect and how to respond to them.
1CATEGORY 1: Hard Failure (easy to detect)
2 Manifestation: HTTP error, timeout, empty response
3 Examples:
4 - Anthropic API returns 500
5 - API rate limit hit (429)
6 - Network timeout to the API endpoint
7 - Invalid API key (401)
8
9 Detection: standard error monitoring
10 Response: immediate fallback to non-AI flow
11
12CATEGORY 2: Soft Failure (hard to detect)
13 Manifestation: response succeeds but output is wrong/low quality
14 Examples:
15 - Inaccurate PR description
16 - Spec audit that misses violations
17 - Wrong cost analysis
18 - Review that is too permissive or too strict
19
20 Detection: needs evaluation/sampling
21 Response: rollback prompt version or disable feature
22
23CATEGORY 3: Degradation (very hard to detect)
24 Manifestation: quality that drops gradually
25 Examples:
26 - Vendor model update that changes behavior
27 - Prompt drift (the context used is stale)
28 - Cumulative template issues
29
30 Detection: longitudinal quality metrics
31 Response: re-evaluate prompts, model update
32
33CATEGORY 4: Cost Anomaly
34 Manifestation: abnormal cost
35 Examples:
36 - Prompt loop bug
37 - Oversized input that isn't truncated
38 - Unintended model upgrade
39
40 Detection: cost monitoring (Article 17)
41 Response: circuit breaker + prompt fixThe key to this classification: hard failures are easily caught by standard monitoring, but soft failures and degradation need dedicated detection mechanisms — and it’s precisely these two categories that most often slip through until users complain.
18.02 Detection System for AI Failures
To catch all four categories, we need a detector that tracks error rate as well as quality trends. The Detector struct below records each request, computes the error rate per window, and stores quality scores to detect degradation.
1// internal/ai/monitor/detector.go
2
3package monitor
4
5import (
6 "context"
7 "time"
8 "sync"
9)
10
11type FailureType string
12
13const (
14 FailureHard FailureType = "hard_failure"
15 FailureSoft FailureType = "soft_failure"
16 FailureDegradation FailureType = "degradation"
17 FailureCostAnomaly FailureType = "cost_anomaly"
18)
19
20type AIFailureEvent struct {
21 Feature string
22 FailureType FailureType
23 Severity string // P1, P2, P3
24 Description string
25 Timestamp time.Time
26 Context map[string]string
27}
28
29type Detector struct {
30 mu sync.RWMutex
31 errorWindows map[string]*ErrorWindow
32 qualityScores map[string][]float64
33}
34
35type ErrorWindow struct {
36 errors int
37 requests int
38 windowStart time.Time
39 windowDur time.Duration
40}
41
42func (d *Detector) RecordRequest(feature string, err error, qualityScore float64) {
43 d.mu.Lock()
44 defer d.mu.Unlock()
45
46 window := d.getOrCreateWindow(feature)
47 window.requests++
48 if err != nil { window.errors++ }
49
50 if qualityScore > 0 {
51 d.qualityScores[feature] = append(d.qualityScores[feature], qualityScore)
52 // Keep only last 100 scores
53 if len(d.qualityScores[feature]) > 100 {
54 d.qualityScores[feature] = d.qualityScores[feature][1:]
55 }
56 }
57}
58
59func (d *Detector) ErrorRate(feature string) float64 {
60 d.mu.RLock()
61 defer d.mu.RUnlock()
62
63 w := d.errorWindows[feature]
64 if w == nil || w.requests == 0 { return 0 }
65 return float64(w.errors) / float64(w.requests)
66}
67
68func (d *Detector) QualityTrend(feature string) (current, baseline float64) {
69 d.mu.RLock()
70 scores := d.qualityScores[feature]
71 d.mu.RUnlock()
72
73 if len(scores) < 10 { return 0, 0 }
74
75 // Recent 10 vs older scores
76 recent := scores[len(scores)-10:]
77 older := scores[:len(scores)-10]
78
79 return avg(recent), avg(older)
80}QualityTrend is the weapon against degradation: by comparing the 10 most recent scores against the older baseline, we can catch a gradual quality decline that never triggers a single error rate.
18.03 Severity Classification
Not every incident deserves to wake up on-call at 3 a.m. The severity matrix below maps symptoms to P1–P4 levels, complete with response time targets and the automatic actions triggered.
1P1 (Critical — response < 15 minutes):
2 - AI feature causes data corruption or a security issue
3 - Error rate > 50% and impacting a core user journey
4 - Cost spike > 100× baseline
5
6 Who: On-call engineer + Tech Lead + Engineering Manager
7 Auto-action: immediate kill switch
8
9P2 (High — response < 1 hour):
10 - Error rate 20-50%, degraded but not broken
11 - Quality score drop > 40% from baseline
12 - Cost spike 10-100× baseline
13
14 Who: On-call engineer
15 Auto-action: rollback prompt, alert on-call
16
17P3 (Medium — response < 4 hours):
18 - Error rate 5-20%
19 - Quality score drop 20-40%
20 - Cost spike 3-10× baseline
21
22 Who: Feature team
23 Auto-action: alert only, no auto-rollback
24
25P4 (Low — response < 1 day):
26 - Minor quality degradation (< 20%)
27 - Isolated incidents
28 - Cost drift (< 3× baseline)
29
30 Who: Feature team (async)
31 Auto-action: log + weekly reviewThis severity mapping aligns response urgency with actual impact: P1 triggers an automatic kill switch and escalation to three people, while P4 is just logged for a weekly review.
18.04 Auto-Detection with GitHub Actions
Manual detection is too slow; we want a system that checks AI health every 15 minutes and reacts on its own. The workflow below tests the Anthropic API, reads the error rate from Prometheus, determines the severity, and automatically disables AI features on a P1.
1# .github/workflows/ai-health-monitor.yml
2name: AI Health Monitor
3
4on:
5 schedule:
6 - cron: '*/15 * * * *' # Every 15 minutes
7
8jobs:
9 monitor:
10 runs-on: ubuntu-latest
11 env:
12 ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
13 steps:
14 - uses: actions/checkout@v4
15 - uses: actions/setup-go@v5
16 with: { go-version: '1.22' }
17
18 - name: Check AI service health
19 id: health
20 run: |
21 # Test the Anthropic API with a minimal call
22 RESPONSE=$(./scripts/claude-call.sh "Reply with 'OK'" 10 2>/dev/null) || {
23 echo "status=DOWN" >> $GITHUB_OUTPUT
24 echo "anthropic_api=DOWN"
25 exit 0
26 }
27
28 [ "$RESPONSE" == "OK" ] && \
29 echo "status=UP" >> $GITHUB_OUTPUT || \
30 echo "status=DEGRADED" >> $GITHUB_OUTPUT
31
32 - name: Check error rates from Prometheus
33 id: metrics
34 run: |
35 ERROR_RATE=$(curl -sf "https://metrics.santekno.com/api/v1/query" \
36 --data-urlencode "query=rate(ai_requests_total{status='error'}[15m]) / rate(ai_requests_total[15m])" \
37 | jq '.data.result[0].value[1]' | tr -d '"' || echo "0")
38
39 echo "error_rate=$ERROR_RATE" >> $GITHUB_OUTPUT
40
41 SEVERITY="P4"
42 awk "BEGIN {exit ($ERROR_RATE > 0.50) ? 0 : 1}" && SEVERITY="P1"
43 awk "BEGIN {exit ($ERROR_RATE > 0.20) ? 0 : 1}" && SEVERITY="P2"
44 awk "BEGIN {exit ($ERROR_RATE > 0.05) ? 0 : 1}" && SEVERITY="P3"
45 echo "severity=$SEVERITY" >> $GITHUB_OUTPUT
46
47 - name: Auto-respond to P1/P2
48 if: |
49 steps.metrics.outputs.severity == 'P1' ||
50 steps.metrics.outputs.severity == 'P2'
51 run: |
52 SEVERITY="${{ steps.metrics.outputs.severity }}"
53 ERROR_RATE="${{ steps.metrics.outputs.error_rate }}"
54
55 # P1: immediate kill switch
56 if [ "$SEVERITY" == "P1" ]; then
57 kubectl set env deployment/santekno-shop \
58 FF_AI_ORDER_REVIEW=false \
59 FF_AI_PRODUCT_DESC=false \
60 --namespace=production
61 echo "::error::P1 AI incident — all AI features disabled"
62 fi
63
64 # Alert Slack
65 curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
66 -H "Content-Type: application/json" \
67 -d "$(jq -n \
68 --arg sev "$SEVERITY" \
69 --arg rate "$ERROR_RATE" \
70 '{text: (":rotating_light: AI Incident " + $sev + ": error rate " + $rate)}'
71 )"This automation cuts Mean Time to Detect to at most 15 minutes and Mean Time to Respond to nearly zero for P1 — the kill switch fires without waiting for a human to wake up and log in.
18.05 Incident Response Runbook
When an alert fires, on-call needs clear steps, not improvisation. The five-step runbook below guides from detection to recovery, with specific actions per severity level.
1# AI Incident Response Runbook
2
3## Step 1: DETECT (0-5 minutes)
4Alert received via:
5- Slack #engineering-alerts (automated)
6- PagerDuty (P1/P2 only)
7- Grafana alert (error rate threshold)
8
9First questions:
10- Which AI feature is failing?
11- What is the error rate? (check Grafana)
12- Are users impacted? (check customer support queue)
13- Is it total failure or degradation?
14
15## Step 2: CONTAIN (5-15 minutes)
16Depending on severity:
17
18P1: Immediate kill switch
19 kubectl set env deployment/santekno-shop FF_AI_X=false
20
21P2: Rollback prompt version
22 git revert <prompt-change-commit>
23 Pipeline redeploys automatically
24
25P3: Monitor and document
26 No immediate action, watch metrics for 30 min
27
28## Step 3: DIAGNOSE (15-30 minutes)
29Check in order:
301. Anthropic status page: status.anthropic.com
312. API error logs: kubectl logs ... | grep "AI\|anthropic"
323. Cost metrics: did cost spike at the same time?
334. Recent deployments: was there a prompt change?
345. Traffic: unusual traffic pattern?
35
36## Step 4: RESOLVE (30-120 minutes)
37Options:
38A. Anthropic outage: wait + fallback enabled, monitor status page
39B. Bug in prompt: fix + re-deploy prompt
40C. Model behavior change: update prompt for new model behavior
41D. Code bug: fix + deploy
42
43## Step 5: RECOVER
44Re-enable feature flag gradually:
45 5% → 30 min → 25% → 1h → 100%This DETECT → CONTAIN → DIAGNOSE → RESOLVE → RECOVER order ensures containment happens before diagnosis: we stop the bleeding first, then hunt for the root cause calmly.
18.06 Triage Decision Tree
In the middle of an incident, decisions must be fast and consistent. The decision tree below maps symptoms to severity level and initial resolution, reducing the on-call’s cognitive load under high pressure.
1AI Incident Triage:
2
3Is error rate > 20%?
4 YES → P2 or higher
5 Is error rate > 50%?
6 YES → P1 — immediate kill switch
7 NO → P2 — rollback prompt + alert on-call
8
9 NO → Is quality score drop > 30%?
10 YES → P2/P3 — evaluate prompt rollback
11 NO → Is cost spike > 10×?
12 YES → P2 — check for loop bug, disable feature
13 NO → P4 — monitor, no immediate action
14
15Common resolution by failure type:
16 Hard failure (API error) → check API status, enable fallback
17 Soft failure (bad output) → rollback prompt version
18 Degradation → re-evaluate and update prompts
19 Cost anomaly → find loop/oversized input, fix + circuit breakerThis decision tree turns triage from art into procedure: each branch leads to one concrete action, so two different engineers will make the same decision for the same symptoms.
18.07 Fallback Strategy Implementation
The most fundamental principle: every AI call must have a non-AI path that always works. The WithFallback wrapper below attempts AI with a timeout, and automatically switches to the non-AI implementation on failure.
1// internal/ai/fallback.go
2// Graceful degradation when AI is unavailable
3
4package ai
5
6import (
7 "context"
8 "errors"
9 "time"
10)
11
12type WithFallback struct {
13 primary AIClient
14 fallback func(ctx context.Context, input any) (string, error)
15 timeout time.Duration
16}
17
18func (w *WithFallback) Call(ctx context.Context, prompt string, input any) (string, error) {
19 // Attempt primary (AI) with timeout
20 ctx, cancel := context.WithTimeout(ctx, w.timeout)
21 defer cancel()
22
23 result, err := w.primary.Complete(ctx, prompt)
24 if err == nil {
25 return result, nil
26 }
27
28 // Log AI failure
29 logger := zerolog.Ctx(ctx)
30 logger.Warn().Err(err).Msg("AI primary failed, using fallback")
31
32 // Use fallback (non-AI implementation)
33 if w.fallback != nil {
34 return w.fallback(ctx, input)
35 }
36
37 // No fallback available — return error
38 return "", fmt.Errorf("ai call failed and no fallback: %w", err)
39}
40
41// Usage:
42aiOrderReview := &ai.WithFallback{
43 primary: anthropicClient,
44 timeout: 5 * time.Second,
45 fallback: func(ctx context.Context, input any) (string, error) {
46 // Return standard order review without AI insights
47 return "Standard review: order eligible based on status and timing.", nil
48 },
49}The 5-second timeout here is no trivial detail: without a time limit, a single AI call that hangs could hold a user request indefinitely — a fast fallback is far better than slow AI.
18.08 Post-Mortem Template for AI Incidents
An undocumented incident is bound to recur. The post-mortem template below standardizes how we record impact, timeline, root cause, and action items after each AI incident.
1# AI Incident Post-Mortem
2
3## Summary
4**Date:** [date]
5**Duration:** [X minutes/hours]
6**Severity:** [P1/P2/P3]
7**Feature affected:** [feature name]
8
9## Impact
10- **Users affected:** [count or percentage]
11- **Requests failed:** [count]
12- **Estimated cost:** [if cost anomaly]
13
14## Timeline (UTC)
15- 10:00: First alert fired (error rate 23%)
16- 10:03: On-call engineer paged
17- 10:08: Feature flag disabled (containment)
18- 10:15: Root cause identified: vendor model update
19- 10:45: Prompt updated and deployed
20- 11:00: Feature re-enabled at 10%
21- 11:30: Feature at 100%, incident resolved
22
23## Root Cause
24[Specific description of what caused the failure]
25
26## Contributing Factors
27- [Factor 1]
28- [Factor 2]
29
30## Detection
31Alert triggered: [what triggered it and how long after failure started]
32
33## Resolution
34[Steps taken to resolve]
35
36## Action Items
37| Action | Owner | Due Date |
38|--------|-------|---------|
39| Add test case for vendor model change | [name] | [date] |
40| Improve quality monitoring (detect earlier) | [name] | [date] |
41
42## Lessons Learned
43[2-3 key takeaways]The Timeline and Action Items sections are the heart of the post-mortem: the timeline measures how fast we detected and responded, while action items with owner and due date ensure lessons turn into real improvements.
18.09 AI-Generated Post-Mortem
Writing a post-mortem takes time, and AI can speed up the first draft. The script below gives the AI raw incident data and asks for a post-mortem draft in English that is ready to be edited.
1#!/bin/bash
2# scripts/generate-ai-postmortem.sh
3# AI helps write the post-mortem from incident data
4
5INCIDENT_DATA=$(cat << 'DATA'
6Feature: ai-order-review
7Duration: 47 minutes
8Error rate peak: 67%
9Users affected: ~2,300
10Root cause: Anthropic model update changed response format
11Resolution: Updated prompt to match new format
12DATA
13)
14
15./scripts/claude-call.sh "Write a concise incident post-mortem in English.
16
17Incident data:
18$INCIDENT_DATA
19
20Format:
21## Incident Summary
22## Impact
23## Timeline (bullets)
24## Root Cause
25## Actions Taken
26## Lessons Learned (3 points)
27## Action Items (table)
28
29Professional, factual, concise. Max 400 words." 600 > \
30 "docs/postmortems/$(date +%Y%m%d)-ai-incident.md"
31
32echo "Post-mortem generated"The AI here speeds up 80% of the mechanical writing work, but its draft must still be validated by the engineer who handled the incident — facts and action items are a human responsibility, not the AI’s.
18.10 Monitoring Quality Over Time
Soft failures and degradation can only be caught by measuring quality periodically. The quality monitor below samples a portion of requests, evaluates them with a cheap model, and triggers an alert when quality drops from the baseline.
1// internal/ai/monitor/quality.go
2// Track prompt output quality using sampling
3
4package monitor
5
6type QualityMonitor struct {
7 evaluator *Evaluator
8 sampleRate float64 // 0.0-1.0: fraction of requests to evaluate
9 tracker *Tracker
10}
11
12// Evaluate quality with AI (using a separate cheap model)
13func (m *QualityMonitor) EvaluateIfSampled(ctx context.Context, feature, input, output string) {
14 if rand.Float64() > m.sampleRate { return }
15
16 // Use haiku to evaluate haiku output (meta-evaluation)
17 evalPrompt := fmt.Sprintf(`Rate this AI output quality (0-100).
18
19Feature: %s
20Input context: %s
21Output: %s
22
23Score on:
24- Accuracy (0-40): is the output correct?
25- Completeness (0-30): are all required elements present?
26- Format (0-30): is the format correct?
27
28Output ONLY: {"score": 85, "issues": ["issue1"]}`, feature, input[:200], output[:500])
29
30 result, err := m.evaluator.Evaluate(ctx, evalPrompt)
31 if err != nil { return }
32
33 m.tracker.RecordQuality(feature, result.Score)
34
35 // Alert if quality drops significantly
36 current, baseline := m.tracker.QualityTrend(feature)
37 if baseline > 0 && current < baseline*0.7 {
38 m.alertQualityDrop(feature, current, baseline)
39 }
40}The meta-evaluation approach — using haiku to judge haiku output — provides a low-cost quality signal: by sampling only 5-10% of requests, we catch degradation early without doubling the AI cost.
18.11 Vendor Model Update Response
A vendor model change is a common cause of silent degradation. The script below runs the entire prompt test suite against the new model and fails if the pass rate drops below a threshold, preventing regressions from reaching users.
1#!/bin/bash
2# scripts/test-model-update.sh
3# Run after detecting a vendor model update
4
5MODEL="${1:-claude-haiku-4-5-20251001}"
6
7echo "Testing prompts against $MODEL..."
8
9PASS=0
10FAIL=0
11
12for test_file in testdata/prompts/*-test-cases.json; do
13 PROMPT_NAME=$(basename "$test_file" -test-cases.json)
14 echo -n "Testing $PROMPT_NAME: "
15
16 # Run evaluation
17 RESULT=$(go run ./cmd/eval-prompt/ "$PROMPT_NAME" "$test_file" "$MODEL" 2>&1)
18 RATE=$(echo "$RESULT" | grep "Pass rate:" | grep -oP '[0-9.]+' || echo "0")
19
20 if awk "BEGIN {exit ($RATE >= 80) ? 0 : 1}"; then
21 echo "PASS $RATE%"
22 PASS=$((PASS+1))
23 else
24 echo "FAIL $RATE% (< 80%)"
25 FAIL=$((FAIL+1))
26 fi
27done
28
29echo ""
30echo "Results: $PASS pass, $FAIL fail"
31[ "$FAIL" -gt 0 ] && echo "Some prompts need updating for $MODEL" && exit 1
32echo "All prompts compatible with $MODEL"Running this suite proactively — even without an update announcement — detects model drift before users notice it, turning degradation from a surprise into something caught in CI.
18.12 Circuit Breaker Pattern for AI Services
When the AI API starts failing in a row, continuing to call it only makes things worse. The circuit breaker below opens the circuit after the failure threshold is reached and tries to recover automatically through a half-open state.
1// internal/ai/circuit.go
2
3package ai
4
5import (
6 "sync/atomic"
7 "time"
8)
9
10type CircuitBreaker struct {
11 state atomic.Int32 // 0=CLOSED, 1=OPEN, 2=HALF_OPEN
12 failures atomic.Int64
13 lastFailure atomic.Int64 // Unix timestamp
14
15 maxFailures int64
16 resetTimeout time.Duration
17}
18
19const (
20 StateClosed = 0
21 StateOpen = 1
22 StateHalfOpen = 2
23)
24
25func (cb *CircuitBreaker) Call(fn func() error) error {
26 switch cb.state.Load() {
27 case StateOpen:
28 // Check if reset timeout elapsed
29 if time.Since(time.Unix(cb.lastFailure.Load(), 0)) > cb.resetTimeout {
30 cb.state.Store(StateHalfOpen)
31 } else {
32 return errors.New("circuit breaker OPEN: AI service unavailable")
33 }
34 }
35
36 err := fn()
37 if err != nil {
38 failures := cb.failures.Add(1)
39 cb.lastFailure.Store(time.Now().Unix())
40
41 if failures >= cb.maxFailures {
42 cb.state.Store(StateOpen)
43 }
44 return err
45 }
46
47 // Success: reset circuit
48 cb.failures.Store(0)
49 cb.state.Store(StateClosed)
50 return nil
51}The half-open state is the self-healing mechanism: after the reset timeout, the breaker tries a single probe call — if it succeeds the circuit closes again, if it fails it opens right back up without flooding the struggling API.
18.13 Incident Communication Templates
Consistent communication during an incident reduces panic and speculation. The set of Slack templates below standardizes messages for the initial P1 stage, the P2 update, and the incident closure.
1Slack notification templates:
2
3P1 - Initial:
4[P1 AI INCIDENT] Feature `ai-order-review` down
5Error rate: 67% (threshold: 50%)
6Auto-action: kill switch enabled
7On-call: @engineer-name paged
8Next update: 15 minutes
9
10P2 - Update:
11[P2 AI INCIDENT] Update: `ai-order-review`
12Root cause identified: Anthropic model update changed format
13Action: Prompt rollback in progress
14ETA: 20 minutes
15
16RESOLVED:
17[RESOLVED] AI Incident - `ai-order-review`
18Duration: 47 minutes
19Resolution: Prompt updated for new model format
20Feature: re-enabled at 10%, monitoring
21Post-mortem: [link]Templates with a “Next update” field keep stakeholders calm: they know when the next update is coming, so on-call can focus on solving the problem instead of answering repeated questions.
18.14 Chaos Engineering for AI Features
The best way to make sure the playbook works is to test it before a real incident. The chaos program below injects latency, error rate, and quality degradation in a controlled way to verify that the detection and response systems actually work.
1// cmd/ai-chaos/main.go
2// Test incident response without a real incident
3
4package main
5
6import (
7 "context"
8 "fmt"
9 "time"
10)
11
12type ChaosScenario struct {
13 Name string
14 Duration time.Duration
15 Apply func(ctx context.Context)
16 Revert func(ctx context.Context)
17}
18
19var scenarios = []ChaosScenario{
20 {
21 Name: "API Latency Spike",
22 Duration: 5 * time.Minute,
23 Apply: func(ctx context.Context) {
24 // Inject artificial latency via middleware
25 setAILatency(5 * time.Second)
26 },
27 Revert: func(ctx context.Context) {
28 setAILatency(0)
29 },
30 },
31 {
32 Name: "High Error Rate",
33 Duration: 3 * time.Minute,
34 Apply: func(ctx context.Context) {
35 setAIErrorRate(0.30) // 30% errors
36 },
37 Revert: func(ctx context.Context) {
38 setAIErrorRate(0)
39 },
40 },
41 {
42 Name: "Quality Degradation",
43 Duration: 10 * time.Minute,
44 Apply: func(ctx context.Context) {
45 // Return low-quality responses
46 setAIQualityMultiplier(0.4) // 40% normal quality
47 },
48 Revert: func(ctx context.Context) {
49 setAIQualityMultiplier(1.0)
50 },
51 },
52}
53
54func main() {
55 for _, scenario := range scenarios {
56 fmt.Printf("Running chaos: %s\n", scenario.Name)
57 ctx := context.Background()
58
59 scenario.Apply(ctx)
60 time.Sleep(scenario.Duration)
61 scenario.Revert(ctx)
62
63 fmt.Printf("Completed: %s — check if system self-healed\n", scenario.Name)
64 }
65}Chaos tests turn assumptions into evidence: if the “High Error Rate” scenario doesn’t trigger a P2 within 5 minutes, we find the hole in detection during staging, not during a real production incident.
18.15 Learning from Every Incident
The true value of an incident lies in the improvement it produces. The checklist below sets the learning rhythm — from a 24-hour post-mortem to a monthly metric review — so each incident raises the system’s reliability.
1After every AI incident (P1, P2, P3):
2
3Within 24 hours:
4- Post-mortem written (use template 18.08)
5- Action items created with owner and due date
6- Root cause documented
7
8Within 1 week:
9- Action items implemented
10- Test case added (to prevent recurrence)
11- Playbook updated (if there's a gap)
12- Monitoring improved (if detection was late)
13
14Monthly review:
15- How many AI incidents this month? Trend?
16- MTTD (Mean Time to Detect): getting faster?
17- MTTR (Mean Time to Resolve): getting faster?
18- Which features are most incident-prone?This rhythm creates a feedback loop: every incident adds a test case and refines the playbook, so the same class of failure won’t surprise the team a second time.
18.16 AI Incident Metrics
To know whether we’re improving, incidents must be measured. The Prometheus metrics below track the number of incidents, Mean Time to Detect, and Mean Time to Resolve as indicators of operational maturity.
1// Track incident metrics for continuous improvement
2
3var (
4 incidentCount = prometheus.NewCounterVec(
5 prometheus.CounterOpts{Name: "ai_incidents_total"},
6 []string{"severity", "feature", "type"},
7 )
8 mttdSeconds = prometheus.NewHistogram(
9 prometheus.HistogramOpts{
10 Name: "ai_incident_mttd_seconds",
11 Buckets: []float64{60, 300, 900, 1800, 3600},
12 },
13 )
14 mttrSeconds = prometheus.NewHistogram(
15 prometheus.HistogramOpts{
16 Name: "ai_incident_mttr_seconds",
17 Buckets: []float64{300, 900, 1800, 3600, 7200},
18 },
19 )
20)
21
22// Monthly targets:
23// MTTD: < 5 minutes (with automated detection)
24// MTTR: < 30 minutes for P2, < 15 minutes for P1Targets of MTTD < 5 minutes and MTTR < 30 minutes give the team a measurable goal: a histogram trend declining month over month is concrete proof that the investment in detection and playbooks is paying off.
18.17 Prevention Checklist
The best incident is the one that never happens. The checklist below serves as the gate before a new AI feature touches production, ensuring fallback, monitoring, testing, and operations are ready from the start.
1Before deploying a new AI feature to production:
2
3Code quality:
4- Fallback implemented (non-AI path always works)
5- Circuit breaker configured
6- Timeout set (max 5 seconds for user-facing, 30 for CI)
7- Error handling: all API errors logged + graceful
8
9Monitoring:
10- Error rate metric per feature
11- Quality sampling configured
12- Cost alert configured
13- Dashboard panel added
14
15Testing:
16- Unit tests: fallback behavior tested
17- Integration tests: timeout behavior tested
18- Load tests: behavior under high concurrency
19- Chaos tests: behavior during API outage
20
21Operations:
22- Kill switch exists (feature flag)
23- Rollback playbook written
24- On-call knows about the feature
25- Post-mortem template preparedThis checklist moves the cost of reliability upfront: a little discipline before deploy is far cheaper than a production incident that surprises the team and users.
18.18 Vendor Relationship Management
Because AI features depend on a third-party API, managing the vendor relationship is part of reliability. The summary below notes the status page, rate limits, deprecation policy, and best practices for dealing with the Anthropic API.
1Anthropic API reliability considerations:
2
3Status page: status.anthropic.com
4 Subscribe to notifications for proactive alerts
5
6Rate limits:
7 Tier 1: 50K tokens/min
8 Tier 2: 100K tokens/min
9 Tier 3: 400K tokens/min
10 → Know your tier, plan accordingly
11
12Model deprecation:
13 Anthropic typically gives 6+ months notice
14 Subscribe to: developers@anthropic.com notifications
15 Test new models with an evaluation suite before switching
16
17Best practices:
18 - Implement exponential backoff for rate limit (429)
19 - Cache responses to reduce API dependency
20 - Test against multiple model versions
21 - Have a fallback model configuredKnowing your rate limit tier and subscribing to deprecation notifications turns surprises into plans: a model update announced six months ahead is no longer a threat but a scheduled backlog item.
18.19 Integration Tests for AI Reliability
Fallback and retry behavior must be tested like any other code — in CI, not in production. The integration tests below verify that a timeout triggers the fallback and that a rate limit triggers exponential backoff as expected.
1// integration/ai_reliability_test.go
2// go:build integration
3
4func TestAIFeature_APITimeout_FallsBack(t *testing.T) {
5 // Setup: AI API with 100ms timeout (unrealistically short)
6 client := ai.NewClient(ai.WithTimeout(100 * time.Millisecond))
7 handler := NewOrderHandler(client, fallbackHandler)
8
9 // Act: call during simulated high latency
10 ctx := context.Background()
11 result, err := handler.ReviewOrder(ctx, testOrder)
12
13 // Assert: should use fallback, not fail
14 require.NoError(t, err)
15 assert.Equal(t, "standard", result.ReviewType)
16 assert.False(t, result.AIEnhanced)
17}
18
19func TestAIFeature_RateLimit_ExponentialBackoff(t *testing.T) {
20 // Mock: first 3 calls return 429, 4th succeeds
21 mockClient := newMockAPIClient()
22 mockClient.SetResponses(429, 429, 429, 200)
23
24 result, err := callWithRetry(ctx, mockClient, "test prompt")
25 require.NoError(t, err)
26 assert.Equal(t, 4, mockClient.CallCount())
27}Tests like these make reliability a machine-verified contract: the AIEnhanced == false assertion during a timeout ensures the fallback is truly active, not merely hoped to be active.
18.20 Summary: Reliability Engineering for AI Features
AI features require reliability engineering just as rigorous as any other feature — even more so, because of their non-deterministic nature.
Key principles:
- Always have a fallback — an AI failure must degrade smoothly, never break
- Detect fast — automated monitoring with a detection time < 5 minutes
- Respond fast — an instant kill switch and a clear playbook
- Keep learning — every incident produces a concrete improvement
The measure of success: MTTD < 5 minutes, MTTR < 30 minutes, and zero incidents that surprise the team. Teams that treat AI reliability as an afterthought will experience incidents that surprise them; teams that invest upfront will experience incidents that their system handles automatically.
With feature flags, cost monitoring, and an incident playbook in hand, we’re ready to weave everything into a single complete pipeline from specification all the way to AI operations in production.