AI Feature Flags Golang: Gradual Rollout, Kill Switch, A/B Testing
Implement AI feature flags for Go services. Gradual rollout, user segmentation, kill switch, and automatic rollback for AI-powered features in production.
AI Feature Flags: Gradual Rollout for AI Features in Go
AI feature flags are the single most important mechanism for shipping AI features to production without gambling your entire user base at once. AI features have characteristics that differ from ordinary features: they are non-deterministic, depend on an external API, and can produce unexpected output. With feature flags, we can roll out AI features gradually, test them in production with a subset of users, and perform instant rollbacks without redeploying.
In this article we build a complete feature flag system for Golang — from percentage rollout based on consistent hashing, atomic kill switch, circuit breaker, all the way to A/B testing — designed specifically to protect the service from the unique risks of AI features.
16.01 Why AI Features Need Feature Flags
Before diving into the code, it’s important to understand why AI features demand a different rollout mechanism. The comparison below places the deploy cycle of an ordinary feature alongside an AI feature and shows the risk that surfaces when both are treated the same.
1Ordinary feature:
2 Deploy → immediately 100% of users
3 If it fails: deploy rollback (5-10 minutes)
4 Risk: all users exposed to the bug
5
6AI feature:
7 Different characteristics:
8 - LLM API can go down → all users affected
9 - Output quality varies per user/input
10 - Cost can spike unexpectedly
11 - A vendor model update can change behavior
12
13 With feature flags:
14 - Deploy code → AI feature still off
15 - Enable for 5% of users → monitor quality
16 - Ramp up gradually if OK
17 - Instant kill switch if there's a problemThe point is clear: AI features bring four sources of uncertainty that ordinary features don’t have, and feature flags turn deployment from “all or nothing” into a controllable, staged process.
16.02 Feature Flag Architecture for Go
The foundation of this system is a Manager interface that hides the flag implementation details behind a stable contract. The code below defines the flag type, the manager interface, and the simplest environment-variable-based implementation.
1// internal/feature/flags.go
2
3package feature
4
5import (
6 "context"
7 "os"
8 "strconv"
9)
10
11type Flag string
12
13const (
14 // AI features
15 AIOrderReview Flag = "ai_order_review"
16 AIProductDescription Flag = "ai_product_description"
17 AICancelOrderAnalysis Flag = "ai_cancel_order_analysis"
18 AISearchEnhancement Flag = "ai_search_enhancement"
19
20 // Experiment flags
21 ExperimentNewPromptV2 Flag = "exp_new_prompt_v2"
22)
23
24type Manager interface {
25 IsEnabled(ctx context.Context, flag Flag) bool
26 IsEnabledForUser(ctx context.Context, flag Flag, userID string) bool
27 GetVariant(ctx context.Context, flag Flag, userID string) string
28}
29
30// SimpleManager: environment-variable based (simple production-ready)
31type SimpleManager struct {
32 flags map[Flag]bool
33}
34
35func NewSimpleManager() *SimpleManager {
36 return &SimpleManager{
37 flags: map[Flag]bool{
38 AIOrderReview: os.Getenv("FF_AI_ORDER_REVIEW") == "true",
39 AIProductDescription: os.Getenv("FF_AI_PRODUCT_DESC") == "true",
40 AICancelOrderAnalysis: os.Getenv("FF_AI_CANCEL_ANALYSIS") == "true",
41 },
42 }
43}
44
45func (m *SimpleManager) IsEnabled(ctx context.Context, flag Flag) bool {
46 return m.flags[flag]
47}
48
49func (m *SimpleManager) IsEnabledForUser(ctx context.Context, flag Flag, userID string) bool {
50 // Hash userID for consistent assignment
51 hashVal := hashUserID(userID)
52 rolloutPct, exists := m.getRolloutPercentage(flag)
53 if !exists { return m.IsEnabled(ctx, flag) }
54 return float64(hashVal%100) < rolloutPct
55}With the Manager interface as the contract, the handler never knows whether a flag is read from an environment variable, Redis, or a kill switch — we’re free to swap the implementation without touching business code.
16.03 Percentage Rollout Implementation
The key to a staged rollout is consistent hashing: the same user must always be in or always out of the rollout, with no flicker. The implementation below combines a whitelist, blacklist, and SHA-256-based bucketing to determine rollout membership deterministically.
1// internal/feature/rollout.go
2
3package feature
4
5import (
6 "crypto/sha256"
7 "encoding/binary"
8 "sync"
9)
10
11type RolloutConfig struct {
12 mu sync.RWMutex
13 configs map[Flag]*FlagConfig
14}
15
16type FlagConfig struct {
17 Enabled bool
18 Rollout float64 // 0.0-100.0 percent
19 UserList []string // specific users (whitelist)
20 ExcludeList []string // specific users (blacklist)
21}
22
23func (r *RolloutConfig) IsEnabledForUser(flag Flag, userID string) bool {
24 r.mu.RLock()
25 cfg, ok := r.configs[flag]
26 r.mu.RUnlock()
27
28 if !ok || !cfg.Enabled { return false }
29
30 // Check whitelist
31 for _, u := range cfg.UserList {
32 if u == userID { return true }
33 }
34
35 // Check blacklist
36 for _, u := range cfg.ExcludeList {
37 if u == userID { return false }
38 }
39
40 // Rollout by percentage using consistent hashing
41 if cfg.Rollout >= 100.0 { return true }
42 if cfg.Rollout <= 0.0 { return false }
43
44 h := sha256.Sum256([]byte(string(flag) + userID))
45 bucket := float64(binary.BigEndian.Uint64(h[:8]) % 100)
46 return bucket < cfg.Rollout
47}
48
49func hashUserID(userID string) uint64 {
50 h := sha256.Sum256([]byte(userID))
51 return binary.BigEndian.Uint64(h[:8])
52}Because the hash is computed from flag + userID, a single user consistently lands in the same bucket throughout the rollout — so raising the percentage from 5% to 25% only adds users, never removes those already in.
16.04 Redis-Backed Feature Flags for Realtime Updates
Environment variables require a redeploy to change; to update flags in realtime we need a backend like Redis. The code below stores flag configuration in Redis with a short-lived local cache so we don’t hammer Redis on every request.
1// internal/feature/redis.go
2// Feature flags that can be updated without a redeploy
3
4package feature
5
6import (
7 "context"
8 "encoding/json"
9 "fmt"
10
11 "github.com/redis/go-redis/v9"
12)
13
14type RedisManager struct {
15 client *redis.Client
16 cache sync.Map // local cache to reduce Redis load
17}
18
19func NewRedisManager(client *redis.Client) *RedisManager {
20 return &RedisManager{client: client}
21}
22
23const flagKeyPrefix = "feature:flag:"
24
25func (m *RedisManager) IsEnabled(ctx context.Context, flag Flag) bool {
26 // Check local cache first
27 if cached, ok := m.cache.Load(string(flag)); ok {
28 return cached.(bool)
29 }
30
31 val, err := m.client.Get(ctx, flagKeyPrefix+string(flag)).Bool()
32 if err != nil { return false }
33
34 // Cache for 30 seconds
35 m.cache.Store(string(flag), val)
36 go func() {
37 time.Sleep(30 * time.Second)
38 m.cache.Delete(string(flag))
39 }()
40
41 return val
42}
43
44func (m *RedisManager) SetFlag(ctx context.Context, flag Flag, config *FlagConfig) error {
45 data, err := json.Marshal(config)
46 if err != nil { return fmt.Errorf("SetFlag marshal: %w", err) }
47
48 return m.client.Set(ctx, flagKeyPrefix+string(flag), data, 0).Err()
49}The trade-off to be aware of: a 30-second local cache means a flag change takes up to 30 seconds to propagate to all pods — acceptable for a rollout, but too slow for an incident. For that we need the next mechanism.
16.05 Kill Switch Pattern
When an incident hits, we need a way to turn off an AI feature in milliseconds without waiting for the cache to expire or a redeploy. The atomic.Bool-based kill switch below provides an instant disable that is safe to access from many goroutines at once.
1// internal/feature/killswitch.go
2// Instant disable without a redeploy or Redis update
3
4package feature
5
6import "sync/atomic"
7
8type KillSwitch struct {
9 flags map[Flag]*atomic.Bool
10}
11
12func NewKillSwitch(flags ...Flag) *KillSwitch {
13 ks := &KillSwitch{flags: make(map[Flag]*atomic.Bool)}
14 for _, f := range flags {
15 v := &atomic.Bool{}
16 v.Store(true) // start enabled
17 ks.flags[f] = v
18 }
19 return ks
20}
21
22// Kill disables flag instantly (atomic operation)
23func (ks *KillSwitch) Kill(flag Flag) {
24 if v, ok := ks.flags[flag]; ok {
25 v.Store(false)
26 }
27}
28
29// Revive re-enables flag
30func (ks *KillSwitch) Revive(flag Flag) {
31 if v, ok := ks.flags[flag]; ok {
32 v.Store(true)
33 }
34}
35
36func (ks *KillSwitch) IsAlive(flag Flag) bool {
37 if v, ok := ks.flags[flag]; ok {
38 return v.Load()
39 }
40 return false
41}
42
43// Usage: during an incident
44// killSwitch.Kill(feature.AIOrderReview)
45// → instantly disables AI for all requests, no redeploy neededAn atomic.Store operation takes effect instantly for every goroutine in the process, so the kill switch is our fastest line of defense — far faster than a Redis update or a Kubernetes rollout.
16.06 Usage in the Handler
All of the mechanisms above are only useful if the handler actually checks them before calling AI. The cancel-order handler below picks the AI path or the standard path based on the flag, while also providing a graceful fallback when the AI call fails.
1// delivery/http/handler/order_handler.go
2
3func (h *OrderHandler) CancelOrder(c echo.Context) error {
4 ctx := c.Request().Context()
5 userID := getUserID(ctx)
6
7 // Handle cancel order with or without AI analysis
8 if h.flags.IsEnabledForUser(ctx, feature.AICancelOrderAnalysis, userID) {
9 return h.cancelOrderWithAI(c, ctx, userID)
10 }
11 return h.cancelOrderStandard(c, ctx, userID)
12}
13
14func (h *OrderHandler) cancelOrderWithAI(c echo.Context, ctx context.Context, userID string) error {
15 // AI-enhanced: analyze order, predict cancel reason, suggest alternatives
16 analysis, err := h.aiUseCase.AnalyzeCancelRequest(ctx, input)
17 if err != nil {
18 // Graceful fallback: if AI fails, use standard flow
19 h.log.Warn().Err(err).Msg("AI analysis failed, falling back to standard")
20 return h.cancelOrderStandard(c, ctx, userID)
21 }
22
23 // Use AI analysis in the response
24 return c.JSON(http.StatusOK, map[string]any{
25 "cancelled": true,
26 "ai_suggestion": analysis.AlternativeSuggestions,
27 })
28}This pattern enforces the core principle: the standard path is always available as a fallback, so an AI failure degrades response quality rather than breaking the request.
16.07 Gradual Rollout Workflow
With environment-variable-based flags, a staged rollout can be run via kubectl commands without changing code. The command sequence below sketches a four-stage timeline from internal team all the way to 100%, complete with the instant rollback command.
1# Roll out AI order review gradually
2
3# Day 1: Internal team only
4kubectl set env deployment/santekno-shop \
5 FF_AI_ORDER_REVIEW_USERS="user-internal-1,user-internal-2" \
6 --namespace=production
7
8# Day 3: 5% of users (if internal is OK)
9kubectl set env deployment/santekno-shop \
10 FF_AI_ORDER_REVIEW_ROLLOUT="5" \
11 --namespace=production
12
13# Day 7: 25% (if 5% is OK)
14kubectl set env deployment/santekno-shop \
15 FF_AI_ORDER_REVIEW_ROLLOUT="25" \
16 --namespace=production
17
18# Day 14: 100% (if 25% is OK)
19kubectl set env deployment/santekno-shop \
20 FF_AI_ORDER_REVIEW_ROLLOUT="100" \
21 --namespace=production
22
23# Instant rollback if there's a problem:
24kubectl set env deployment/santekno-shop \
25 FF_AI_ORDER_REVIEW_ROLLOUT="0" \
26 --namespace=productionEach percentage increase is a decision gate: we only advance to the next stage after the metrics at the previous stage are healthy, and a single ROLLOUT="0" command is enough to pull the feature entirely.
16.08 A/B Test for AI Features
Gradual rollout answers “is this feature safe?”, while A/B testing answers “is this feature actually better?”. The tracker below records the outcome and metrics per variant so we can compare control versus treatment quantitatively.
1// internal/feature/experiment.go
2// Track A/B experiment results
3
4package feature
5
6type ExperimentTracker struct {
7 mu sync.Mutex
8 results map[string]*ExperimentResult
9}
10
11type ExperimentResult struct {
12 FlagName string
13 Variant string // "control" or "treatment"
14 UserID string
15 Timestamp time.Time
16 Outcome string // e.g., "completed", "cancelled", "abandoned"
17 MetricValue float64 // e.g., time_to_complete_ms, satisfaction_score
18}
19
20func (t *ExperimentTracker) Record(result ExperimentResult) {
21 t.mu.Lock()
22 defer t.mu.Unlock()
23
24 key := result.FlagName + ":" + result.Variant
25 if _, ok := t.results[key]; !ok {
26 t.results[key] = &ExperimentResult{}
27 }
28 // Aggregate metrics...
29}
30
31// Usage:
32// After cancel order with AI analysis:
33tracker.Record(feature.ExperimentResult{
34 FlagName: "ai_cancel_analysis",
35 Variant: "treatment", // or "control" if no AI
36 UserID: userID,
37 Outcome: "completed",
38 MetricValue: float64(time.Since(start).Milliseconds()),
39})With outcome data per variant, the promote-to-100% decision stops being a guess and becomes a numbers-based conclusion — for example, treatment raising completion rate by 4% even though latency is slightly higher.
16.09 Circuit Breaker for AI Features
Beyond manual control, we want AI features to disable themselves when the error rate spikes. The atomic-counter-based circuit breaker below counts errors within a time window and “trips” automatically once the threshold is exceeded.
1// internal/feature/circuit.go
2// Automatically disable an AI feature if error rate is high
3
4package feature
5
6import (
7 "sync/atomic"
8 "time"
9)
10
11type CircuitBreaker struct {
12 flag Flag
13 maxErrors int64
14 windowSecs int64
15 manager Manager
16
17 errors atomic.Int64
18 lastReset atomic.Int64
19 tripped atomic.Bool
20}
21
22func (cb *CircuitBreaker) RecordError() {
23 now := time.Now().Unix()
24 lastReset := cb.lastReset.Load()
25
26 // Reset counter if the window has elapsed
27 if now - lastReset > cb.windowSecs {
28 cb.errors.Store(0)
29 cb.lastReset.Store(now)
30 cb.tripped.Store(false)
31 }
32
33 newCount := cb.errors.Add(1)
34
35 if newCount >= cb.maxErrors && !cb.tripped.Load() {
36 cb.tripped.Store(true)
37 // Notify: AI feature auto-disabled due to high error rate
38 log.Error().
39 Str("flag", string(cb.flag)).
40 Int64("errors", newCount).
41 Msg("Circuit breaker tripped — AI feature disabled")
42 }
43}
44
45func (cb *CircuitBreaker) IsOpen() bool {
46 return cb.tripped.Load()
47}The circuit breaker complements the kill switch: the kill switch is a manual intervention when we’re aware of a problem, while the circuit breaker is an automatic defense for problems we haven’t seen yet.
16.10 Feature Flag API for Admin
So the operations team can change flags without kubectl access, we expose an internal admin API. The handler below validates input, enforces admin authentication, then writes the flag configuration to the manager.
1// delivery/http/handler/admin/feature_handler.go
2// Internal API to manage feature flags
3
4func (h *FeatureAdminHandler) SetFlag(c echo.Context) error {
5 var req struct {
6 Flag string `json:"flag"`
7 Enabled bool `json:"enabled"`
8 Rollout float64 `json:"rollout"` // 0-100
9 }
10
11 if err := c.Bind(&req); err != nil {
12 return c.JSON(http.StatusBadRequest, map[string]string{"error": "INVALID_REQUEST"})
13 }
14
15 // Require admin auth
16 if !isAdmin(c.Request().Context()) {
17 return c.JSON(http.StatusForbidden, map[string]string{"error": "FORBIDDEN"})
18 }
19
20 if err := h.manager.SetFlag(c.Request().Context(), feature.Flag(req.Flag), &feature.FlagConfig{
21 Enabled: req.Enabled,
22 Rollout: req.Rollout,
23 }); err != nil {
24 return c.JSON(http.StatusInternalServerError, map[string]string{"error": "INTERNAL_ERROR"})
25 }
26
27 return c.JSON(http.StatusOK, map[string]bool{"success": true})
28}The isAdmin gate here is no formality: because this API can turn AI features on or off in production, it must be protected as tightly as any other operational endpoint.
16.11 Monitoring Feature Flag Usage
Without metrics, a rollout is flying blind. The Prometheus instrumentation below counts usage and errors per flag by variant and user segment, so every rollout decision has evidence behind it.
1// Prometheus metrics for feature flag usage
2
3var flagUsage = prometheus.NewCounterVec(
4 prometheus.CounterOpts{Name: "feature_flag_usage_total"},
5 []string{"flag", "variant", "user_segment"},
6)
7
8var flagErrors = prometheus.NewCounterVec(
9 prometheus.CounterOpts{Name: "feature_flag_errors_total"},
10 []string{"flag"},
11)
12
13// Inside the feature manager:
14func (m *Manager) IsEnabledForUser(ctx context.Context, flag Flag, userID string) bool {
15 enabled := m.check(flag, userID)
16
17 variant := "disabled"
18 if enabled { variant = "enabled" }
19
20 flagUsage.WithLabelValues(string(flag), variant, getUserSegment(userID)).Inc()
21
22 return enabled
23}The variant and user_segment labels let us slice the data per group — for example, comparing the error rate of premium users versus new users before expanding the rollout.
16.12 Testing Feature Flag Behavior
Behavior that depends on a flag must be testable deterministically, not depend on the environment. The tests below use a mock flag manager to force the flag on/off and verify that the AI path is used or skipped as expected.
1// In unit tests: explicitly enable/disable flags
2
3type MockFlagManager struct {
4 flags map[feature.Flag]bool
5}
6
7func (m *MockFlagManager) IsEnabled(ctx context.Context, flag feature.Flag) bool {
8 return m.flags[flag]
9}
10
11func (m *MockFlagManager) IsEnabledForUser(ctx context.Context, flag feature.Flag, userID string) bool {
12 return m.flags[flag]
13}
14
15// Usage in a test:
16func (s *CancelOrderSuite) TestCancelOrderWithAI_AIEnabled_UsesAIAnalysis() {
17 // Arrange: AI enabled
18 s.flagManager.flags[feature.AICancelOrderAnalysis] = true
19 s.mockAI.EXPECT().AnalyzeCancel(...).Return(analysis, nil)
20
21 // Act
22 result, err := s.handler.CancelOrder(ctx, input)
23
24 // Assert
25 s.NoError(err)
26 s.NotNil(result.AISuggestions)
27}
28
29func (s *CancelOrderSuite) TestCancelOrderWithAI_AIDisabled_SkipsAI() {
30 // Arrange: AI disabled
31 s.flagManager.flags[feature.AICancelOrderAnalysis] = false
32 // AI mock NOT called (gomock verifies this)
33
34 // Act & Assert
35 _, err := s.handler.CancelOrder(ctx, input)
36 s.NoError(err)
37}These two tests lock in the most important contract: when the flag is off, the AI mock must not be called at all — gomock fails the test if it is, ensuring the fallback path is truly isolated.
16.13 Deployment Pipeline with Feature Flags
The safe principle is: deploy code with all AI flags defaulting off, verify it’s healthy, then enable gradually. The pipeline snippet below turns that principle into explicit deployment steps.
1# Deploy with feature flags disabled by default
2
3- name: Deploy with AI features off
4 run: |
5 kubectl set env deployment/santekno-shop \
6 FF_AI_ORDER_REVIEW=false \
7 FF_AI_PRODUCT_DESC=false \
8 FF_AI_CANCEL_ANALYSIS=false \
9 --namespace=production
10
11- name: Verify deployment healthy (no AI)
12 run: ./scripts/smoke-test.sh santekno.com
13
14- name: Enable AI for internal team
15 if: success()
16 run: |
17 # Enable only for the internal user list (testing)
18 kubectl set configmap ai-features \
19 --from-literal=ai_order_review_users="internal-user-1,internal-user-2" \
20 --namespace=productionBy separating “deploy code” from “enable feature”, we make sure the smoke test runs under no-AI-load conditions first — if there’s a non-AI regression, it’s detected before AI features add another variable.
16.14 Feature Flag Best Practices
After all the components are built, there’s a set of practical rules that distinguish a healthy flag system from one that becomes technical debt. The DO/DON’T list below summarizes practices proven in production.
1DO:
2- Default every AI flag to false (opt-in, not opt-out)
3- Consistent hashing for user assignment (deterministic)
4- Circuit breaker to auto-disable on high error rate
5- Metrics for each flag state (enabled/disabled per user segment)
6- Audit log for every flag change
7- Gradual rollout: 5% → 25% → 50% → 100%
8
9DON'T:
10- Store PII in flag config (use a user ID hash, not email)
11- Flags that are never cleaned up (technical debt)
12- Enable a flag in production without monitoring
13- A kill switch that requires a redeploy
14- A/B tests without a sample size calculationThe common thread of all these rules: flags must default safe (off), be deterministic, be monitored, and have a retirement plan — flags left to pile up without cleanup add the very risk they were supposed to reduce.
16.15 Flag Lifecycle Management
Feature flags have a lifecycle, and ignoring their final stage (cleanup) is a common mistake. The lifecycle scheme below maps a flag’s journey from development to removal, plus a periodic audit schedule.
1Flag lifecycle:
21. DEVELOPMENT: flag created, default false, code behind the flag
32. INTERNAL_TESTING: enabled for the internal team
43. CANARY: 5-10% rollout, monitor metrics
54. GENERAL_AVAILABILITY: 100% rollout, flag still present
65. CLEANUP: remove flag and old code path (once stable for 30 days)
7
8Schedule a quarterly flag audit:
9 - Flags at 100% rollout > 30 days: schedule cleanup
10 - Flags still at 0% > 60 days: are they still relevant?
11 - Flags without monitoring: add metricsThe quarterly audit keeps the system lean: a flag that has been 100% stable for 30 days should be removed together with its old code path, because every live flag adds cognitive load when reading the code.
16.16 Config Management for Feature Flags
So the flag configuration stays consistent and easy to validate, we load it from the environment using a tagged struct. The code below defines the config structure along with safe defaults and its loader.
1// internal/config/flags.go
2// Load feature flag config from the environment
3
4type FeatureFlagConfig struct {
5 AIOrderReview struct {
6 Enabled bool `env:"FF_AI_ORDER_REVIEW" envDefault:"false"`
7 Rollout float64 `env:"FF_AI_ORDER_REVIEW_ROLLOUT" envDefault:"0"`
8 UserList []string `env:"FF_AI_ORDER_REVIEW_USERS" envSeparator:","`
9 }
10 AIProductDesc struct {
11 Enabled bool `env:"FF_AI_PRODUCT_DESC" envDefault:"false"`
12 Rollout float64 `env:"FF_AI_PRODUCT_DESC_ROLLOUT" envDefault:"0"`
13 }
14}
15
16func LoadFeatureFlagConfig() (*FeatureFlagConfig, error) {
17 var cfg FeatureFlagConfig
18 if err := envconfig.Process("", &cfg); err != nil {
19 return nil, fmt.Errorf("LoadFeatureFlagConfig: %w", err)
20 }
21 return &cfg, nil
22}Notice the envDefault:"false" and envDefault:"0" on each AI flag — safe defaults are enforced at the struct level, so forgetting to set an environment variable means the AI feature stays off, not on uncontrolled.
16.17 Canary Analysis with AI
Interestingly, AI itself can help decide whether a canary is worth promoting. The script below collects control and treatment metrics, then asks the AI to recommend PROMOTE, MAINTAIN, or ROLLBACK.
1#!/bin/bash
2# After the canary period, AI analyzes whether the feature is ready for full rollout
3
4FLAG="${1}"
5CANARY_PCT="${2}"
6HOURS="${3:-24}"
7
8# Collect metrics for the canary period
9CONTROL_METRICS=$(curl -sf "https://metrics.santekno.com/api/v1/query_range" \
10 --data-urlencode "query=rate(api_requests_total{feature_flag=\"${FLAG}\",variant=\"control\"}[1h])")
11
12TREATMENT_METRICS=$(curl -sf "https://metrics.santekno.com/api/v1/query_range" \
13 --data-urlencode "query=rate(api_requests_total{feature_flag=\"${FLAG}\",variant=\"treatment\"}[1h])")
14
15./scripts/claude-call.sh "Analyze A/B test results for feature flag '$FLAG'.
16
17Canary: $CANARY_PCT% traffic, $HOURS hours observation.
18
19Control metrics: $CONTROL_METRICS
20Treatment metrics: $TREATMENT_METRICS
21
22Should we:
23A) PROMOTE to 100% (treatment better or equal)
24B) MAINTAIN at $CANARY_PCT% (need more data)
25C) ROLLBACK (treatment clearly worse)
26
27Output: DECISION: [A/B/C]
28Rationale: [2-3 sentences]" 200The AI here is not the final decision-maker but an analysis assistant: it condenses metric signals into a structured recommendation that an engineer must still validate before the rollout is raised.
16.18 Feature Flag Dashboard
For quick visibility, a simple dashboard that exposes the status of all flags is a huge help to on-call. The handler below serves the status of every flag in a JSON format easy for both tooling and humans to consume.
1// cmd/feature-dashboard/main.go
2// Simple dashboard for the status of all feature flags
3
4func main() {
5 http.HandleFunc("/flags", func(w http.ResponseWriter, r *http.Request) {
6 flags := getAllFlags()
7
8 w.Header().Set("Content-Type", "application/json")
9 json.NewEncoder(w).Encode(map[string]any{
10 "timestamp": time.Now(),
11 "flags": flags,
12 })
13 })
14
15 log.Fatal(http.ListenAndServe(":8091", nil))
16}
17
18// Output:
19// {
20// "timestamp": "2026-07-20T10:00:00Z",
21// "flags": {
22// "ai_order_review": {"enabled": true, "rollout": 25.0},
23// "ai_product_desc": {"enabled": false, "rollout": 0.0},
24// "ai_cancel_analysis": {"enabled": true, "rollout": 100.0}
25// }
26// }A single /flags endpoint like this becomes the source of truth during an incident: instead of guessing each pod’s configuration, on-call just looks at the current flag state in one view.
16.19 Integration with Spec Kit
A flag-gated feature still needs a clear specification — including its behavior when the flag is on, off, and when AI fails. The example spec below writes three acceptance criteria that cover all three conditions.
1# Spec for an AI-gated feature
2# .specify/features/ai-order-review/spec.md
3
4## Feature: AI Order Review
5
6### Acceptance Criteria
7
8**AC1: AI enabled → enhanced review**
9- Given: feature flag `ai_order_review` enabled for this user
10- When: GET /orders/:id
11- Then: response includes `ai_insights` field
12
13**AC2: AI disabled → standard review**
14- Given: feature flag `ai_order_review` disabled for this user
15- When: GET /orders/:id
16- Then: response does NOT include `ai_insights` field
17- Then: response is identical to the pre-AI version
18
19**AC3: AI API failure → graceful fallback**
20- Given: feature flag enabled
21- When: AI service returns error
22- Then: response identical to AC2 (standard review)
23- Then: error logged with contextAC3 is the most crucial: the spec explicitly requires that an AI failure produces a response identical to the non-AI version — making “fallback mandatory” a testable contract rather than mere good intention.
16.20 Summary: Feature Flags as the AI Safety Net
Feature flags are the safety net that lets us ship AI features with confidence:
- Deploy code without exposing the feature to users
- Enable gradually with monitoring between stages
- Instant rollback without a deployment via the kill switch
- A/B test to validate business impact
- Circuit breaker to auto-disable when the error rate spikes
Key principle: AI features must always have a fallback to non-AI behavior, and that fallback must be the default. Every mechanism in this article — from consistent hashing to the circuit breaker — ultimately serves one goal: making sure an AI failure never becomes a user failure.
Once AI features can be turned on safely, the next question is how much they cost — and how to keep the LLM bill from quietly exploding.