Multi-Environment Deployment Golang CI: AI Checkpoint, Progressive Delivery
Set up multi-environment deployment for a Go service with AI checkpoints. Progressive delivery from staging to production with automated verification and rollback.
Multi-Environment Deployment with AI Checkpoints
Multi-environment deployment in Golang with AI checkpoints is the safest way to release a Go service to production without a knot in your stomach every time you hit the deploy button. Deploying to production has always been the most critical moment in software delivery — get it wrong once and users feel the impact immediately.
In this article we build a deployment pipeline that is progressive and safe: from staging to production, with AI checkpoints that verify each stage before promoting changes to the next environment. The idea is simple — the further a change moves toward production, the more evidence we gather that the deployment is safe.
13.1 Multi-Environment Architecture
The foundation of a safe deployment is a clear chain of environments, where each stage adds another gate before a change reaches users. The diagram below shows the environment chain we use at Santekno Shop, from the developer machine all the way to production.
1Santekno Shop environment chain:
2
3DEVELOPMENT (local)
4 Developer machine
5 Docker Compose
6 No pipeline
7
8 ↓ PR merge to develop
9
10STAGING
11 Environment: staging.santekno.com
12 Deploy: automatic after merge to develop
13 Database: staging DB (synthetic data)
14 Kafka: staging cluster
15
16 ↓ AI health check + smoke test
17 ↓ Manual approval (optional, sprint demo)
18
19PRODUCTION
20 Environment: santekno.com
21 Deploy: manual trigger or automatic after staging passes
22 Database: production DB
23 Kafka: production cluster
24
25 ↓ Progressive rollout (canary)
26 ↓ AI health monitoring
27 ↓ Rollback if neededNotice that no change can “jump” straight to production — every environment is a gate that must be cleared first, and that is exactly what makes this progressive delivery safe.
13.2 Staging Deployment Workflow
Staging is the first mandatory gate, and deploying to it must be fully automated so there is no friction. The GitHub Actions workflow below builds the image, runs the tests, deploys to the Kubernetes staging namespace, and then triggers an AI health check as a checkpoint.
1# .github/workflows/staging-deploy.yml
2name: Deploy to Staging
3
4on:
5 push:
6 branches: [develop]
7
8env:
9 GO_VERSION: '1.22'
10 REGISTRY: ghcr.io
11 IMAGE_NAME: ${{ github.repository }}
12 ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
13
14jobs:
15 build:
16 name: Build & Push Image
17 runs-on: ubuntu-latest
18 permissions:
19 packages: write
20 contents: read
21 outputs:
22 image_tag: ${{ steps.tag.outputs.tag }}
23
24 steps:
25 - uses: actions/checkout@v4
26
27 - name: Set image tag
28 id: tag
29 run: |
30 TAG="${{ github.sha }}"
31 echo "tag=$TAG" >> $GITHUB_OUTPUT
32
33 - name: Build and test
34 run: |
35 go test -race ./...
36 go build -o /tmp/app ./cmd/server/
37
38 - name: Build Docker image
39 run: |
40 docker build \
41 --label "git.sha=${{ github.sha }}" \
42 --label "git.branch=develop" \
43 --label "build.timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
44 -t "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.tag.outputs.tag }}" \
45 -t "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:staging-latest" \
46 .
47
48 - name: Push image
49 uses: docker/login-action@v3
50 with:
51 registry: ${{ env.REGISTRY }}
52 username: ${{ github.actor }}
53 password: ${{ secrets.GITHUB_TOKEN }}
54
55 deploy-staging:
56 name: Deploy to Staging
57 runs-on: ubuntu-latest
58 needs: build
59 environment:
60 name: staging
61 url: https://staging.santekno.com
62 permissions:
63 deployments: write
64 steps:
65 - uses: actions/checkout@v4
66
67 - name: Deploy to staging
68 env:
69 KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }}
70 run: |
71 kubectl set image deployment/santekno-shop \
72 api="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.build.outputs.image_tag }}" \
73 --namespace=staging
74 kubectl rollout status deployment/santekno-shop \
75 --namespace=staging \
76 --timeout=120s
77
78 - name: Run database migrations
79 env:
80 DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}
81 run: |
82 go run ./cmd/migrate/main.go up
83
84 staging-health-check:
85 name: Staging Health Check
86 runs-on: ubuntu-latest
87 needs: deploy-staging
88 steps:
89 - uses: actions/checkout@v4
90
91 - name: Wait for services to stabilize
92 run: sleep 30
93
94 - name: Run smoke tests
95 id: smoke
96 run: |
97 chmod +x scripts/smoke-test.sh
98 RESULTS=$(./scripts/smoke-test.sh staging.santekno.com)
99 echo "results=$RESULTS" >> $GITHUB_OUTPUT
100
101 - name: AI Health Analysis
102 id: ai_health
103 continue-on-error: true
104 env:
105 ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
106 run: |
107 chmod +x scripts/ai-deployment-check.sh
108 ANALYSIS=$(./scripts/ai-deployment-check.sh \
109 "staging.santekno.com" \
110 "${{ needs.build.outputs.image_tag }}")
111 echo "analysis=$ANALYSIS" >> $GITHUB_OUTPUT
112 VERDICT=$(echo "$ANALYSIS" | grep "VERDICT:" | grep -oP '(HEALTHY|DEGRADED|FAILING)' || echo "HEALTHY")
113 echo "verdict=$VERDICT" >> $GITHUB_OUTPUT
114
115 - name: Fail if staging is unhealthy
116 run: |
117 if [ "${{ steps.ai_health.outputs.verdict }}" == "FAILING" ]; then
118 echo "::error::Staging deployment is FAILING — investigate before production"
119 exit 1
120 fiThe key to this workflow is the strict separation between build, deploy-staging, and staging-health-check — so that if the AI checkpoint returns a FAILING verdict, the pipeline stops before any change makes its way to production.
13.3 Smoke Test Script
Before the AI analyzes anything, we need a deterministic verification that the deployment actually works. The smoke test script below checks the health endpoint, the API version, database and Redis connectivity, and response time — all via simple HTTP calls.
1#!/bin/bash
2# scripts/smoke-test.sh
3# Basic smoke tests to verify the deployment succeeded
4
5set -euo pipefail
6
7HOST="${1:-staging.santekno.com}"
8BASE_URL="https://${HOST}"
9PASS=true
10RESULTS=()
11
12# Test 1: Health check endpoint
13HEALTH=$(curl -sf "${BASE_URL}/health" 2>/dev/null) || {
14 RESULTS+=("Health check: FAIL")
15 PASS=false
16}
17
18if [ -n "$HEALTH" ]; then
19 STATUS=$(echo "$HEALTH" | jq -r '.status' 2>/dev/null || echo "unknown")
20 if [ "$STATUS" == "ok" ]; then
21 RESULTS+=("Health check: OK")
22 else
23 RESULTS+=("Health check: $STATUS")
24 PASS=false
25 fi
26fi
27
28# Test 2: API version endpoint
29VERSION=$(curl -sf "${BASE_URL}/api/v1/version" 2>/dev/null | jq -r '.version' 2>/dev/null || echo "")
30if [ -n "$VERSION" ]; then
31 RESULTS+=("API version: $VERSION")
32else
33 RESULTS+=("API version: FAIL")
34 PASS=false
35fi
36
37# Test 3: Database connectivity (via health endpoint)
38DB_STATUS=$(curl -sf "${BASE_URL}/health" | jq -r '.checks.database' 2>/dev/null || echo "unknown")
39[ "$DB_STATUS" == "ok" ] && RESULTS+=("Database: connected") || RESULTS+=("Database: $DB_STATUS")
40
41# Test 4: Redis connectivity
42REDIS_STATUS=$(curl -sf "${BASE_URL}/health" | jq -r '.checks.redis' 2>/dev/null || echo "unknown")
43[ "$REDIS_STATUS" == "ok" ] && RESULTS+=("Redis: connected") || RESULTS+=("Redis: $REDIS_STATUS")
44
45# Test 5: Response time check
46RESPONSE_TIME=$(curl -sf -w "%{time_total}" -o /dev/null "${BASE_URL}/api/v1/orders" \
47 -H "Authorization: Bearer ${SMOKE_TEST_TOKEN:-}" 2>/dev/null || echo "999")
48
49if awk "BEGIN {exit ($RESPONSE_TIME < 2.0) ? 0 : 1}"; then
50 RESULTS+=("Response time: ${RESPONSE_TIME}s")
51else
52 RESULTS+=("Response time slow: ${RESPONSE_TIME}s")
53fi
54
55# Output
56echo "=== Smoke Test Results: $HOST ==="
57for result in "${RESULTS[@]}"; do
58 echo " $result"
59done
60
61[ "$PASS" == "true" ] && echo "SMOKE_STATUS: PASS" || echo "SMOKE_STATUS: FAIL"A good smoke test does not stop at /health — it exercises real business flows (listing orders, response time), because that is what users actually experience, not just whether the binary is alive.
13.4 AI Deployment Health Check
Deterministic metrics catch the obvious problems, but there are patterns in the logs that slip past numeric thresholds. The script below collects the health response, the metrics, and the most recent logs, then hands them to the AI for semantic analysis and a HEALTHY | DEGRADED | FAILING verdict.
1#!/bin/bash
2# scripts/ai-deployment-check.sh
3# AI analyzes deployment health from multiple data points
4
5set -euo pipefail
6
7HOST="${1}"
8IMAGE_TAG="${2:-unknown}"
9BASE_URL="https://${HOST}"
10
11# Collect metrics
12HEALTH=$(curl -sf "${BASE_URL}/health" 2>/dev/null || echo '{"status":"error"}')
13METRICS=$(curl -sf "${BASE_URL}/metrics" 2>/dev/null | head -50 || echo "metrics unavailable")
14LOGS=$(kubectl logs deployment/santekno-shop --namespace=staging \
15 --since=5m --tail=50 2>/dev/null || echo "logs unavailable")
16
17PROMPT="Analyze deployment health for Santekno Shop Go service.
18
19Environment: Staging
20Image: $IMAGE_TAG
21Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)
22
23HEALTH CHECK RESPONSE:
24$HEALTH
25
26RECENT LOGS (last 5 minutes):
27$LOGS
28
29METRICS:
30$METRICS
31
32Analyze:
331. Are there any ERROR or PANIC log entries?
342. Is the health check reporting all services healthy?
353. Are there unusual patterns in the metrics?
364. Any signs of memory leak, connection pool exhaustion, or high error rate?
37
38Output:
39VERDICT: HEALTHY | DEGRADED | FAILING
40Issues (if any):
41 - [issue]: [description and severity]
42RECOMMENDATION: [next steps if not HEALTHY]"
43
44RESULT=$(./scripts/claude-call.sh "$PROMPT" 500) || {
45 echo "AI analysis unavailable"
46 echo "VERDICT: HEALTHY" # graceful: assume healthy if AI unavailable
47 exit 0
48}
49
50echo "$RESULT"The important detail in this script is the graceful degradation pattern: if the AI is unavailable, the verdict defaults to HEALTHY so that tooling downtime never blocks a deployment that is actually healthy.
13.5 Production Deployment: Progressive Rollout
Once staging passes, promotion to production is not done at 100% all at once — we use a canary. The workflow below deploys to a small subset of pods, monitors the error rate for a few minutes, then asks the AI to assess whether it is safe to promote fully.
1# .github/workflows/production-deploy.yml
2name: Deploy to Production
3
4on:
5 workflow_dispatch:
6 inputs:
7 image_tag:
8 description: 'Image tag to deploy (default: staging-latest)'
9 required: false
10 default: 'staging-latest'
11 rollout_percentage:
12 description: 'Canary rollout percentage (10, 25, 50, 100)'
13 required: false
14 default: '10'
15
16env:
17 ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
18
19jobs:
20 pre-deploy-check:
21 name: Pre-Production Checks
22 runs-on: ubuntu-latest
23 steps:
24 - uses: actions/checkout@v4
25
26 - name: Verify staging is healthy
27 run: |
28 chmod +x scripts/smoke-test.sh
29 ./scripts/smoke-test.sh staging.santekno.com || {
30 echo "::error::Staging is not healthy — cannot proceed to production"
31 exit 1
32 }
33
34 canary-deploy:
35 name: Canary Deploy (${{ github.event.inputs.rollout_percentage }}%)
36 runs-on: ubuntu-latest
37 needs: pre-deploy-check
38 environment:
39 name: production
40 url: https://santekno.com
41 permissions:
42 deployments: write
43 steps:
44 - uses: actions/checkout@v4
45
46 - name: Deploy canary
47 env:
48 KUBECONFIG: ${{ secrets.PRODUCTION_KUBECONFIG }}
49 IMAGE_TAG: ${{ github.event.inputs.image_tag }}
50 ROLLOUT_PCT: ${{ github.event.inputs.rollout_percentage }}
51 run: |
52 TOTAL_REPLICAS=$(kubectl get deployment santekno-shop \
53 -n production -o jsonpath='{.spec.replicas}')
54 CANARY_REPLICAS=$(( TOTAL_REPLICAS * ROLLOUT_PCT / 100 ))
55 [ "$CANARY_REPLICAS" -lt 1 ] && CANARY_REPLICAS=1
56 echo "Deploying to $CANARY_REPLICAS/$TOTAL_REPLICAS pods ($ROLLOUT_PCT%)"
57 kubectl set image deployment/santekno-shop \
58 api="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${IMAGE_TAG}" \
59 --namespace=production
60 kubectl rollout status deployment/santekno-shop \
61 --namespace=production --timeout=180s
62
63 - name: Monitor canary (5 minutes)
64 run: |
65 for i in {1..5}; do
66 sleep 60
67 echo "Minute $i/5 check..."
68 ERROR_RATE=$(curl -sf "https://metrics.santekno.com/api/v1/query?query=rate(http_requests_errors[1m])" \
69 2>/dev/null | jq '.data.result[0].value[1]' 2>/dev/null || echo "0")
70 echo " Error rate: $ERROR_RATE"
71 if awk "BEGIN {exit ($ERROR_RATE > 0.05) ? 0 : 1}"; then
72 echo "::error::Error rate $ERROR_RATE > 5% — triggering rollback"
73 kubectl rollout undo deployment/santekno-shop --namespace=production
74 exit 1
75 fi
76 done
77 echo "Canary monitoring passed"
78
79 - name: AI canary assessment
80 id: canary_check
81 continue-on-error: true
82 run: |
83 chmod +x scripts/ai-deployment-check.sh
84 ANALYSIS=$(./scripts/ai-deployment-check.sh "santekno.com" "${{ github.event.inputs.image_tag }}")
85 VERDICT=$(echo "$ANALYSIS" | grep "VERDICT:" | grep -oP '(HEALTHY|DEGRADED|FAILING)' || echo "HEALTHY")
86 echo "verdict=$VERDICT" >> $GITHUB_OUTPUT
87
88 - name: Promote or rollback
89 run: |
90 VERDICT="${{ steps.canary_check.outputs.verdict }}"
91 if [ "$VERDICT" == "FAILING" ]; then
92 echo "::error::Canary is FAILING — rolling back"
93 kubectl rollout undo deployment/santekno-shop --namespace=production
94 exit 1
95 elif [ "$VERDICT" == "DEGRADED" ]; then
96 echo "::warning::Canary is DEGRADED — review before full promotion"
97 else
98 echo "Canary healthy — ready for full promotion"
99 fiThis canary pattern combines two signals: a deterministic error-rate threshold (instant auto-rollback) and a semantic AI verdict — so the promote/rollback decision stands on both quantitative and qualitative evidence.
13.6 Automated Rollback
If metrics cross the safe boundary, the system must be able to roll back without waiting for a human. The script below checks the error rate and P99 latency from Prometheus, and as soon as a threshold is exceeded, it immediately runs kubectl rollout undo and then notifies Slack.
1#!/bin/bash
2# scripts/auto-rollback.sh
3# Trigger a rollback based on metrics thresholds
4
5set -euo pipefail
6
7ENVIRONMENT="${1:-production}"
8NAMESPACE="${ENVIRONMENT}"
9
10MAX_ERROR_RATE=0.05 # 5%
11MAX_P99_LATENCY=2000 # 2 seconds in ms
12
13TRIGGER_ROLLBACK=false
14REASON=""
15
16ERROR_RATE=$(curl -sf "https://metrics.santekno.com/api/v1/query" \
17 --data-urlencode "query=rate(http_requests_total{status=~'5..'}[2m])/rate(http_requests_total[2m])" \
18 2>/dev/null | jq '.data.result[0].value[1]' | tr -d '"' || echo "0")
19
20if awk "BEGIN {exit ($ERROR_RATE > $MAX_ERROR_RATE) ? 0 : 1}"; then
21 TRIGGER_ROLLBACK=true
22 REASON="Error rate ${ERROR_RATE} > ${MAX_ERROR_RATE} threshold"
23fi
24
25P99_LATENCY=$(curl -sf "https://metrics.santekno.com/api/v1/query" \
26 --data-urlencode "query=histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[2m])) * 1000" \
27 2>/dev/null | jq '.data.result[0].value[1]' | tr -d '"' || echo "0")
28
29if awk "BEGIN {exit ($P99_LATENCY > $MAX_P99_LATENCY) ? 0 : 1}"; then
30 TRIGGER_ROLLBACK=true
31 REASON="P99 latency ${P99_LATENCY}ms > ${MAX_P99_LATENCY}ms threshold"
32fi
33
34if [ "$TRIGGER_ROLLBACK" == "true" ]; then
35 echo "ROLLBACK TRIGGERED: $REASON"
36 kubectl rollout undo deployment/santekno-shop --namespace="$NAMESPACE"
37 kubectl rollout status deployment/santekno-shop --namespace="$NAMESPACE" --timeout=120s
38 echo "Rollback completed"
39 curl -X POST "$SLACK_WEBHOOK" \
40 -H "Content-Type: application/json" \
41 -d "$(jq -n --arg reason "$REASON" --arg env "$ENVIRONMENT" \
42 '{text: ("Auto-rollback triggered in *" + $env + "*\nReason: " + $reason)}')"
43 exit 1
44fi
45
46echo "Deployment healthy, no rollback needed"This threshold-based automatic rollback is the last safety net — when something really goes wrong, the service recovers within minutes, with no engineer having to be woken up in the middle of the night.
13.7 Informative Deployment Notifications
A finished deployment must be communicated, but a generic “deploy succeeded” notification doesn’t help much. The step below uses AI to compose a concise, contextual Slack notification in Indonesian, whatever the final status turns out to be.
1- name: Send deployment notification
2 if: always()
3 env:
4 ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
5 run: |
6 STATUS="${{ job.status }}"
7 VERSION="${{ github.event.inputs.image_tag }}"
8 ENVIRONMENT="production"
9 ACTOR="${{ github.actor }}"
10
11 PROMPT="Generate a brief deployment notification (max 100 words, Bahasa Indonesia).
12 Deploy: $VERSION
13 Environment: $ENVIRONMENT
14 Status: $STATUS
15 Deployed by: $ACTOR
16 Time: $(date -u '+%Y-%m-%d %H:%M UTC')
17 Format sebagai Slack notification yang informative tapi concise."
18
19 NOTIFICATION=$(./scripts/claude-call.sh "$PROMPT" 150) || \
20 NOTIFICATION="Deploy $VERSION ke $ENVIRONMENT: $STATUS"
21
22 curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
23 -H "Content-Type: application/json" \
24 -d "$(jq -n --arg text "$NOTIFICATION" '{text: $text}')"With if: always() and a fallback string, the notification is always sent — even when the deploy fails or the AI is unavailable — so the team never loses visibility into what happened in production.
13.8 Feature Flags for Safe Deployment
Code deployment and feature activation should be decoupled, and feature flags are the mechanism. The Go code below shows a simple flag manager backed by environment variables, then used in a handler to choose the new or the old flow at runtime.
1// internal/feature/flags.go
2// Feature flags for gradual rollout without deployment
3
4package feature
5
6import (
7 "context"
8 "os"
9)
10
11type Flag string
12
13const (
14 NewCancelOrderFlow Flag = "new_cancel_order_flow"
15 FlashSaleFeature Flag = "flash_sale_feature"
16 AIProductDescription Flag = "ai_product_description"
17)
18
19type Manager struct {
20 // Can be backed by Redis, LaunchDarkly, or simple env vars
21 flags map[Flag]bool
22}
23
24func NewManager() *Manager {
25 return &Manager{
26 flags: map[Flag]bool{
27 NewCancelOrderFlow: os.Getenv("FF_NEW_CANCEL_ORDER") == "true",
28 FlashSaleFeature: os.Getenv("FF_FLASH_SALE") == "true",
29 AIProductDescription: os.Getenv("FF_AI_PRODUCT_DESC") == "true",
30 },
31 }
32}
33
34func (m *Manager) IsEnabled(ctx context.Context, flag Flag) bool {
35 return m.flags[flag]
36}
37
38// Usage in a handler:
39func (h *OrderHandler) CancelOrder(c echo.Context) error {
40 if h.flags.IsEnabled(c.Request().Context(), feature.NewCancelOrderFlow) {
41 return h.newCancelOrderFlow(c)
42 }
43 return h.legacyCancelOrderFlow(c)
44}With feature flags you can deploy new feature code to production in a disabled state, then enable it gradually without a redeploy — reducing risk and making it easy to roll a feature back.
13.9 Deployment Checklist Generator
Every release carries a different risk depending on its changes, so a generic checklist is often irrelevant. The script below collects the commits and the number of migrations since the last deploy, then asks the AI to build a deployment checklist specific to those changes.
1#!/bin/bash
2# scripts/generate-deployment-checklist.sh
3# AI generates a deployment checklist based on the changes
4
5set -euo pipefail
6
7VERSION="${1}"
8ENVIRONMENT="${2:-staging}"
9
10LAST_DEPLOY_TAG=$(git tag -l "deploy-${ENVIRONMENT}-*" | sort -V | tail -1)
11COMMITS=$(git log "${LAST_DEPLOY_TAG}..HEAD" --no-merges --pretty=format:"%s" 2>/dev/null | head -20 || \
12 git log --no-merges --pretty=format:"%s" -20)
13
14MIGRATION_COUNT=$(git diff "${LAST_DEPLOY_TAG}..HEAD" --name-only -- 'migrations/**' 2>/dev/null | wc -l || echo "0")
15
16PROMPT="Generate deployment checklist untuk release $VERSION ke $ENVIRONMENT.
17
18Changes (commit messages):
19$COMMITS
20
21Migration count: $MIGRATION_COUNT
22
23Create a concise deployment checklist in Bahasa Indonesia:
241. Pre-deployment steps berdasarkan tipe perubahan
252. Deployment steps
263. Post-deployment verification steps
274. Rollback triggers (apa yang perlu di-monitor)
28
29Be specific berdasarkan changes. Skip generic steps yang tidak relevant."
30
31./scripts/claude-call.sh "$PROMPT" 500A checklist generated from a real diff is far more useful than a static template — it highlights the actual risks (for example, that a migration is present) instead of generic steps that get skipped because they feel irrelevant.
13.10 Tips & Gotchas
Before moving on to more advanced topics, there are a few practical lessons worth remembering when building a pipeline like this.
Tip 1: Smoke tests must be specific. A test that only checks /health is not enough. Test a real business flow: can you create an order? can you list products? is the response time acceptable?
Tip 2: Start the canary with a small percentage. Begin with 10% of traffic. If 10% is healthy for 5 minutes, promote to 25%, then 50%, then 100%. Don’t jump straight to 100%.
Tip 3: Realistic auto-rollback thresholds. A 5% error rate may be too low for a canary that only sees a little traffic. Calibrate the threshold against the baseline error rate in production.
Tip 4: AI health checks as a supplement, not a replacement. The AI can miss things. Pair it with deterministic metric checks (error rate, latency, CPU).
Gotcha 1: Deploying during peak hours. AI notifications are not enough — timing needs human judgment. Don’t auto-deploy to production during busy hours (12–14 WIB for e-commerce).
Gotcha 2: Migrations and rollback. If a deployment is rolled back but the migration already ran, the data schema may be incompatible with the old version. Plan for this carefully.
Gotcha 3: Flaky smoke tests. A smoke test that fails intermittently will block every deployment. Fix flaky tests before turning them into a deployment gate.
The bottom line: no matter how sophisticated the tooling is, it still needs calibration and human judgment at the critical points — automation extends your capacity, it doesn’t remove your responsibility.
13.11 GitOps: Environment Configuration as Code
The differences between environments should live in git, not hidden in a dashboard. The example values below show how staging and production configuration are separated as files that can be reviewed through a PR.
1# environments/staging/values.yaml
2image: { tag: staging-latest }
3replicas: 2
4env:
5 LOG_LEVEL: debug
6 FEATURE_FLAG_CANCEL_ORDER: "true"
7
8# environments/production/values.yaml
9image: { tag: v1.5.0 } # pinned version
10replicas: 5
11env:
12 LOG_LEVEL: info
13 FEATURE_FLAG_CANCEL_ORDER: "false" # off until fully testedWith configuration as code, every environment change goes through PR review and is tracked in git history — an automatic audit trail for the question “why is production set up like this?”.
13.12 Deployment Duration Tracking
A deployment that keeps getting slower is an early signal of pipeline trouble. The short script below measures deployment duration, warns if it crosses a threshold, and sends the number as a metric for trend analysis.
1#!/bin/bash
2# scripts/track-deployment-duration.sh
3
4START_TIME=$(date +%s)
5
6# ... deployment steps run here ...
7
8END_TIME=$(date +%s)
9DURATION=$((END_TIME - START_TIME))
10echo "Deployment duration: ${DURATION}s"
11
12[ "$DURATION" -gt 300 ] && echo "::warning::Slow deployment (${DURATION}s > 300s)"
13
14curl -sX POST "https://metrics.santekno.com/api/v1/events" \
15 -H "Content-Type: application/json" \
16 -d "{\"event\": \"deployment\", \"env\": \"$1\", \"duration\": $DURATION}" || trueTracking deployment duration over time helps detect pipeline degradation early — before a “3-minute deploy” quietly turns into a “15-minute deploy”.
13.13 Zero-Downtime Strategy
A safe deployment means users never see an error while pods are being replaced. The rolling update configuration below, paired with graceful shutdown in Go, ensures no old pod is killed before a new pod is ready to serve.
1# Kubernetes rolling update strategy
2strategy:
3 type: RollingUpdate
4 rollingUpdate:
5 maxUnavailable: 0 # zero downtime: no pod removed before new one ready
6 maxSurge: 1 # one extra pod during update
7
8# Coupled with graceful shutdown in Go (main.go):
9# signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
10# <-quit
11# ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
12# defer cancel()
13# srv.Shutdown(ctx) // wait up to 30s for in-flight requestsThe combination of maxUnavailable: 0 and a 30-second graceful shutdown guarantees that in-flight requests finish before the old pod is killed — the foundation of a zero-downtime deployment.
13.14 Deployment Audit Trail
For audit purposes, every production deployment must leave a trace. The step below records deployment metadata (timestamp, version, actor, smoke test result, AI verdict) as an artifact retained for months.
1- name: Record deployment
2 run: |
3 cat > /tmp/deploy-record.json << EOF
4 {
5 "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
6 "environment": "production",
7 "version": "${{ github.ref_name }}",
8 "actor": "${{ github.actor }}",
9 "sha": "${{ github.sha }}",
10 "smoke_test": "${{ steps.smoke.outcome }}",
11 "ai_verdict": "${{ steps.ai_health.outputs.verdict }}"
12 }
13 EOF
14
15- uses: actions/upload-artifact@v4
16 with:
17 name: deploy-record-${{ github.sha }}
18 path: /tmp/deploy-record.json
19 retention-days: 90An audit trail like this answers the important questions when tracing an incident: “which version was active at the time, who deployed it, and what was the health check verdict?”.
13.15 Environment-Specific Config Management
Per-environment configuration must be validated before deploy, not after the service crashes. The directory structure and validation commands below ensure every required field is filled and no secret is empty before a change goes up.
1# configs/ organized per environment with separate secrets
2configs/
3 base/ # shared config values
4 staging/ # staging-specific overrides
5 production/ # production-specific overrides (secrets via vault)
6
7# Validation: make sure the config is valid before deploy
8go run ./cmd/validate-config/main.go --env staging
9go run ./cmd/validate-config/main.go --env production
10# Checks: required fields, type validation, no empty secretsValidating configuration as a pipeline step prevents the most infuriating class of bug: a service that deploys successfully but crashes immediately because one secret was forgotten.
13.16 Summary: Progressive Confidence Building
Multi-environment deployment with AI checkpoints is, at its core, about building confidence gradually — each environment provides more evidence that a change is safe to promote.
Key elements:
- Staging as a mandatory gate before production
- Smoke tests for functional verification
- AI health checks for semantic analysis of logs and metrics
- Gradual canary rollout (10% → 25% → 50% → 100%)
- Auto-rollback based on metrics thresholds
The philosophy: humans decide when to promote, the system monitors and rolls back automatically. With this pattern, deployment stops being a nerve-wracking event and becomes a layered process where every stage can be trusted.