Skip to content
Santekno.com | Level Up Your Engineering Skills
EN
📖 0%
19 Nov 2026 · 22 min read ·Article 92 / 208
Go

Golang Kafka Integration: Outbox Worker, Producer, Consumer Group E-Commerce

A complete Kafka integration for Santekno Shop. An outbox worker that publishes to Kafka, producer setup with sarama, and a consumer group for notifications — an end-to-end event flow.

IH
Ihsan Arif
Writer at Santekno · Backend Engineer

Golang Kafka integration with the outbox pattern is the backbone of Santekno Shop’s event-driven architecture that we haven’t yet completed. In Part 2, the outbox worker was still a stub — events were written to the outbox_events table but never actually published to Kafka. In this article we close that loop: a Kafka producer that publishes events from the outbox, a consumer group for the notification service, and an end-to-end test for the entire event flow.

We start from topic setup, a publisher with an idempotent producer, an outbox worker based on FOR UPDATE SKIP LOCKED, all the way to the partition key strategy for preserving event ordering.


12.01 Kafka Topics Setup

The first step is creating all the topics Santekno Shop needs. The script below creates the order, payment, and DLQ topics with 3 partitions each.

bash
 1# Setup Kafka topics for Santekno Shop
 2# Run after kafka is ready in docker-compose
 3
 4KAFKA_BROKER=localhost:9092
 5
 6# Create all topics with 3 partitions, replication factor 1 (dev)
 7topics=(
 8    "santekno.orders.placed"
 9    "santekno.orders.confirmed"
10    "santekno.orders.cancelled"
11    "santekno.orders.shipped"
12    "santekno.payments.completed"
13    "santekno.payments.failed"
14    "santekno.notifications.dlq"
15)
16
17for topic in "${topics[@]}"; do
18    kafka-topics --create \
19        --bootstrap-server $KAFKA_BROKER \
20        --topic "$topic" \
21        --partitions 3 \
22        --replication-factor 1 \
23        --if-not-exists
24    echo "Created topic: $topic"
25done
26
27# Verify
28kafka-topics --list --bootstrap-server $KAFKA_BROKER | grep santekno

Choosing 3 partitions per topic isn’t a random number: it sets the maximum number of consumer instances that can process in parallel, so plan it according to your target throughput from the start.


12.02 Kafka Publisher: Interface and Implementation

The publisher wraps sarama’s SyncProducer with a configuration that prioritizes durability and exactly-once per partition. The code below shows the full implementation plus a NoOpPublisher for development without Kafka.

go
 1// internal/event/kafka/publisher.go
 2
 3package kafka
 4
 5import (
 6    "context"
 7    "encoding/json"
 8    "fmt"
 9
10    "github.com/IBM/sarama"
11
12    "github.com/santekno/santekno-shop/internal/event"
13)
14
15type Publisher struct {
16    producer sarama.SyncProducer
17}
18
19func NewPublisher(brokers []string) (*Publisher, error) {
20    config := sarama.NewConfig()
21    config.Producer.Return.Successes = true
22    config.Producer.Return.Errors   = true
23    config.Producer.RequiredAcks    = sarama.WaitForAll  // acks from all ISR
24    config.Producer.Retry.Max       = 3
25    config.Producer.Compression     = sarama.CompressionSnappy
26
27    // Idempotent producer: exactly-once per partition
28    config.Producer.Idempotent = true
29    config.Net.MaxOpenRequests = 1 // required for idempotent
30
31    producer, err := sarama.NewSyncProducer(brokers, config)
32    if err != nil {
33        return nil, fmt.Errorf("kafkaPublisher.New: %w", err)
34    }
35
36    return &Publisher{producer: producer}, nil
37}
38
39func (p *Publisher) Publish(ctx context.Context, e event.Event) error {
40    payload, err := json.Marshal(e.Payload)
41    if err != nil {
42        return fmt.Errorf("kafkaPublisher.Publish: marshal: %w", err)
43    }
44
45    msg := &sarama.ProducerMessage{
46        Topic: e.Topic,
47        Value: sarama.ByteEncoder(payload),
48    }
49
50    // Use event key for partition routing (same key = same partition = ordered)
51    if e.Key != "" {
52        msg.Key = sarama.StringEncoder(e.Key)
53    }
54
55    _, _, err = p.producer.SendMessage(msg)
56    if err != nil {
57        return fmt.Errorf("kafkaPublisher.Publish [%s]: %w", e.Topic, err)
58    }
59
60    return nil
61}
62
63func (p *Publisher) Close() error {
64    return p.producer.Close()
65}
66
67// NoOpPublisher: for development without Kafka running
68type NoOpPublisher struct{}
69
70func (p *NoOpPublisher) Publish(_ context.Context, e event.Event) error {
71    fmt.Printf("[EVENT] topic=%s key=%s\n", e.Topic, e.Key)
72    return nil
73}

The combination of Idempotent = true and RequiredAcks = WaitForAll is the key: the first prevents duplicates caused by network retries, the second guarantees data isn’t lost until every ISR replica has confirmed.


12.03 Outbox Worker: Full Implementation

The outbox worker is the bridge between PostgreSQL (ACID) and Kafka (eventually consistent). The code below polls unpublished events, sends them to Kafka, then marks them published.

go
  1// internal/event/outbox/worker.go
  2
  3package outbox
  4
  5import (
  6    "context"
  7    "encoding/json"
  8    "fmt"
  9    "time"
 10
 11    "github.com/jackc/pgx/v5/pgxpool"
 12    "github.com/santekno/santekno-shop/internal/event"
 13)
 14
 15type Worker struct {
 16    db        *pgxpool.Pool
 17    publisher event.Publisher
 18    batchSize int
 19    interval  time.Duration
 20    stopCh    chan struct{}
 21}
 22
 23func NewWorker(db *pgxpool.Pool, publisher event.Publisher) *Worker {
 24    return &Worker{
 25        db:        db,
 26        publisher: publisher,
 27        batchSize: 100,
 28        interval:  5 * time.Second,
 29        stopCh:    make(chan struct{}),
 30    }
 31}
 32
 33func (w *Worker) Run(ctx context.Context) {
 34    ticker := time.NewTicker(w.interval)
 35    defer ticker.Stop()
 36
 37    fmt.Println("Outbox worker started")
 38    for {
 39        select {
 40        case <-ctx.Done():
 41            fmt.Println("Outbox worker stopped (context)")
 42            return
 43        case <-w.stopCh:
 44            fmt.Println("Outbox worker stopped (signal)")
 45            return
 46        case <-ticker.C:
 47            if err := w.processOnce(ctx); err != nil {
 48                fmt.Printf("outbox worker error: %v\n", err)
 49            }
 50        }
 51    }
 52}
 53
 54func (w *Worker) Stop() {
 55    close(w.stopCh)
 56}
 57
 58// processOnce: take a batch of unpublished events, publish to Kafka
 59func (w *Worker) processOnce(ctx context.Context) error {
 60    // Fetch unpublished events (SKIP LOCKED for multiple worker instances)
 61    rows, err := w.db.Query(ctx,
 62        `SELECT id, topic, payload, event_type
 63         FROM outbox_events
 64         WHERE published = false
 65         ORDER BY created_at ASC
 66         LIMIT $1
 67         FOR UPDATE SKIP LOCKED`,  // prevent duplicate processing
 68        w.batchSize,
 69    )
 70    if err != nil {
 71        return fmt.Errorf("outbox.processOnce: fetch: %w", err)
 72    }
 73    defer rows.Close()
 74
 75    type outboxRow struct {
 76        ID        string
 77        Topic     string
 78        Payload   json.RawMessage
 79        EventType string
 80    }
 81
 82    var events []outboxRow
 83    for rows.Next() {
 84        var row outboxRow
 85        if err := rows.Scan(&row.ID, &row.Topic, &row.Payload, &row.EventType); err != nil {
 86            return fmt.Errorf("outbox.processOnce: scan: %w", err)
 87        }
 88        events = append(events, row)
 89    }
 90    rows.Close()
 91
 92    if len(events) == 0 {
 93        return nil // nothing to do
 94    }
 95
 96    // Publish each event
 97    published := 0
 98    for _, e := range events {
 99        if err := w.publisher.Publish(ctx, event.Event{
100            Topic:   e.Topic,
101            Key:     extractKey(e.Payload), // extract order_id or user_id
102            Payload: e.Payload,
103        }); err != nil {
104            // Log but don't stop — try the next event
105            fmt.Printf("outbox: publish failed [%s]: %v\n", e.EventType, err)
106            continue
107        }
108
109        // Mark as published
110        if _, err := w.db.Exec(ctx,
111            `UPDATE outbox_events
112             SET published = true, published_at = NOW()
113             WHERE id = $1`,
114            e.ID,
115        ); err != nil {
116            fmt.Printf("outbox: mark published failed [%s]: %v\n", e.ID, err)
117        }
118        published++
119    }
120
121    if published > 0 {
122        fmt.Printf("outbox: published %d/%d events\n", published, len(events))
123    }
124    return nil
125}
126
127// extractKey: extract the partition key from the JSON payload
128func extractKey(payload json.RawMessage) string {
129    var m map[string]interface{}
130    if err := json.Unmarshal(payload, &m); err != nil {
131        return ""
132    }
133    for _, key := range []string{"order_id", "user_id", "payment_id"} {
134        if val, ok := m[key].(string); ok && val != "" {
135            return val
136        }
137    }
138    return ""
139}

The FOR UPDATE SKIP LOCKED clause in the fetch query is what makes the worker safe to run multi-instance: each worker locks the rows it takes and skips rows already locked by another worker, so no event gets published twice.


12.04 Consumer Group: Order Event Consumer

This generic consumer group maps topics to handlers and applies manual commit. The code below is a reusable version of the consumer that can be used across services.

go
 1// internal/event/kafka/consumer_group.go
 2
 3package kafka
 4
 5import (
 6    "context"
 7    "fmt"
 8    "log"
 9
10    "github.com/IBM/sarama"
11)
12
13type MessageHandler func(ctx context.Context, topic string, key, value []byte) error
14
15type ConsumerGroup struct {
16    client   sarama.ConsumerGroup
17    topics   []string
18    handlers map[string]MessageHandler
19}
20
21func NewConsumerGroup(brokers []string, groupID string, handlers map[string]MessageHandler) (*ConsumerGroup, error) {
22    config := sarama.NewConfig()
23    config.Consumer.Group.Rebalance.GroupStrategies = []sarama.BalanceStrategy{
24        sarama.NewBalanceStrategyRoundRobin(),
25    }
26    config.Consumer.Offsets.Initial      = sarama.OffsetNewest
27    config.Consumer.Offsets.AutoCommit.Enable = false // manual commit only
28
29    client, err := sarama.NewConsumerGroup(brokers, groupID, config)
30    if err != nil {
31        return nil, fmt.Errorf("kafka.NewConsumerGroup: %w", err)
32    }
33
34    topics := make([]string, 0, len(handlers))
35    for topic := range handlers {
36        topics = append(topics, topic)
37    }
38
39    return &ConsumerGroup{
40        client:   client,
41        topics:   topics,
42        handlers: handlers,
43    }, nil
44}
45
46func (cg *ConsumerGroup) Run(ctx context.Context) error {
47    handler := &groupHandler{handlers: cg.handlers}
48    for {
49        if err := cg.client.Consume(ctx, cg.topics, handler); err != nil {
50            log.Printf("consumer group error: %v", err)
51        }
52        if ctx.Err() != nil {
53            return ctx.Err()
54        }
55    }
56}
57
58func (cg *ConsumerGroup) Close() error {
59    return cg.client.Close()
60}
61
62type groupHandler struct {
63    handlers map[string]MessageHandler
64}
65
66func (h *groupHandler) Setup(sarama.ConsumerGroupSession) error   { return nil }
67func (h *groupHandler) Cleanup(sarama.ConsumerGroupSession) error { return nil }
68
69func (h *groupHandler) ConsumeClaim(
70    session sarama.ConsumerGroupSession,
71    claim sarama.ConsumerGroupClaim,
72) error {
73    for msg := range claim.Messages() {
74        handler, ok := h.handlers[msg.Topic]
75        if !ok {
76            session.MarkMessage(msg, "")
77            continue
78        }
79
80        ctx := context.Background()
81        if err := handler(ctx, msg.Topic, msg.Key, msg.Value); err != nil {
82            // Don't commit — Kafka will redeliver
83            log.Printf("handler error [%s]: %v", msg.Topic, err)
84            continue
85        }
86
87        // Commit only on success
88        session.MarkMessage(msg, "")
89        session.Commit()
90    }
91    return nil
92}

The MessageHandler type as a function signature makes this consumer agnostic to the domain — any service can register its own handler without changing the core consumer group code.


12.05 Wiring: Main Server with Kafka

All the components are assembled in the main server, with Kafka optional via a config flag. The code below unifies the publisher, outbox worker, consumer, and HTTP server in a single lifecycle.

go
 1// cmd/server/main.go — updated with Kafka
 2
 3func main() {
 4    cfg, _ := config.Load()
 5    ctx, cancel := context.WithCancel(context.Background())
 6    defer cancel()
 7
 8    // Database
 9    pool, _ := pgxpool.New(ctx, cfg.Database.URL)
10    defer pool.Close()
11    rdb := redis.NewClient(&redis.Options{Addr: cfg.Redis.URL})
12    defer rdb.Close()
13
14    // Kafka Publisher
15    var publisher event.Publisher
16    if cfg.Kafka.Enabled {
17        kafkaPublisher, err := kafkaPkg.NewPublisher(cfg.Kafka.Brokers)
18        if err != nil {
19            log.Fatalf("kafka publisher: %v", err)
20        }
21        defer kafkaPublisher.Close()
22        publisher = kafkaPublisher
23    } else {
24        publisher = &event.NoOpPublisher{} // development without Kafka
25    }
26
27    // Build container (all usecases)
28    container := BuildContainer(cfg, pool, rdb, publisher)
29
30    // Outbox Worker
31    outboxWorker := outbox.NewWorker(pool, publisher)
32    go outboxWorker.Run(ctx)
33
34    // Kafka Consumer (notification service)
35    if cfg.Kafka.Enabled {
36        notifHandlers := buildNotificationHandlers(pool)
37        consumer, _ := kafkaPkg.NewConsumerGroup(
38            cfg.Kafka.Brokers,
39            "santekno-notification",
40            notifHandlers,
41        )
42        defer consumer.Close()
43        go consumer.Run(ctx)
44    }
45
46    // HTTP server
47    e := SetupRouter(container, cfg)
48    go e.Start(":" + cfg.App.Port)
49
50    // Graceful shutdown
51    quit := make(chan os.Signal, 1)
52    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
53    <-quit
54
55    outboxWorker.Stop()
56    shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
57    defer shutdownCancel()
58    e.Shutdown(shutdownCtx)
59    cancel() // stop consumer goroutines
60}

The cfg.Kafka.Enabled flag lets Kafka be disabled in development and replaced by NoOpPublisher — developers can run the application without having to spin up a local Kafka cluster.


12.06 Config: Kafka Settings

The Kafka configuration is read from environment variables with sensible defaults. The code below defines KafkaConfig and how it’s loaded.

go
 1// config/config.go
 2
 3type KafkaConfig struct {
 4    Enabled    bool
 5    Brokers    []string
 6    GroupID    string
 7}
 8
 9func Load() (*Config, error) {
10    return &Config{
11        // ...
12        Kafka: KafkaConfig{
13            Enabled: getEnvBool("KAFKA_ENABLED", true),
14            Brokers: strings.Split(getEnv("KAFKA_BROKERS", "localhost:9092"), ","),
15            GroupID: getEnv("KAFKA_GROUP_ID", "santekno-notification"),
16        },
17    }, nil
18}
19
20// .env.example
21// KAFKA_ENABLED=true
22// KAFKA_BROKERS=localhost:9092
23// KAFKA_GROUP_ID=santekno-notification

Keeping the brokers and group ID in environment variables keeps the application portable across environments — from a developer’s laptop to a production cluster — without changing code.


12.07 End-to-End Test: PlaceOrder → Kafka Event

The end-to-end test proves the entire flow works: an order created via API, published by the worker, and received in Kafka. The code below runs that scenario with a direct sarama consumer.

go
 1// tests/e2e/order_flow_test.go
 2//go:build e2e
 3
 4package e2e_test
 5
 6import (
 7    "context"
 8    "encoding/json"
 9    "testing"
10    "time"
11
12    "github.com/IBM/sarama"
13    "github.com/stretchr/testify/assert"
14    "github.com/stretchr/testify/require"
15)
16
17// TestOrderFlow_PlaceOrder_PublishesKafkaEvent:
18// 1. Place an order via HTTP API
19// 2. Wait for the outbox worker to publish
20// 3. Verify the event is received in Kafka
21func TestOrderFlow_PlaceOrder_PublishesKafkaEvent(t *testing.T) {
22    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
23    defer cancel()
24
25    // Setup Kafka consumer to capture events
26    config := sarama.NewConfig()
27    config.Consumer.Offsets.Initial = sarama.OffsetNewest
28    consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, config)
29    require.NoError(t, err)
30    defer consumer.Close()
31
32    partConsumer, err := consumer.ConsumePartition("santekno.orders.placed", 0, sarama.OffsetNewest)
33    require.NoError(t, err)
34    defer partConsumer.Close()
35
36    // Place order via API (using test helper)
37    orderID := placeTestOrder(t)
38
39    // Wait for the outbox worker to publish (max 15 seconds)
40    select {
41    case msg := <-partConsumer.Messages():
42        var payload map[string]interface{}
43        require.NoError(t, json.Unmarshal(msg.Value, &payload))
44
45        assert.Equal(t, orderID, payload["order_id"])
46        assert.NotEmpty(t, payload["user_id"])
47        assert.NotEmpty(t, payload["total_amount_idr"])
48        t.Logf("Event received: order_id=%s", orderID)
49
50    case err := <-partConsumer.Errors():
51        t.Fatalf("consumer error: %v", err)
52
53    case <-ctx.Done():
54        t.Fatal("timeout: event not received within 15 seconds")
55    }
56}
57
58func placeTestOrder(t *testing.T) string {
59    t.Helper()
60    // Login, place order via HTTP, return order_id
61    // (simplified — actual HTTP calls to the test server)
62    return "test-order-id-xxx"
63}

This test verifies the system’s most important contract: an order created by a customer really does produce an event in Kafka within seconds — something an isolated unit test can’t prove.


12.08 FOR UPDATE SKIP LOCKED: Concurrency Safety

To understand why FOR UPDATE SKIP LOCKED is crucial, let’s compare the scenario with and without it. The diagram below illustrates how two workers could process the same event without the lock.

text
 1PROBLEM: multiple outbox worker instances running
 2
 3Without SKIP LOCKED:
 4  Worker A: SELECT WHERE published=false LIMIT 100
 5  Worker B: SELECT WHERE published=false LIMIT 100
 6  → Both take the same events!
 7  → Duplicate publish to Kafka
 8
 9WITH FOR UPDATE SKIP LOCKED:
10  Worker A: SELECT ... FOR UPDATE SKIP LOCKED
11  → Lock rows, skip those already locked by another worker
12  Worker B: SELECT ... FOR UPDATE SKIP LOCKED
13  → Skip rows locked by Worker A
14  → Take rows not yet locked
15
16Result: each event is processed by only one worker
17Note: the Kafka consumer must still be idempotent (Kafka itself can redeliver)

Note the last remark: SKIP LOCKED prevents duplicates on the producer side, but Kafka can still redeliver on the consumer side — that’s why idempotency in the handler (via the event_id from Article 10) remains mandatory.


12.09 Kafka Offset Management

The offset commit strategy determines whether the system loses messages on a crash. The code below contrasts the dangerous auto-commit with the safe manual commit.

go
 1// The correct offset commit strategy for the notification service
 2
 3// WRONG: AutoCommit enabled
 4// → Message committed before it's processed → message loss on crash
 5
 6config.Consumer.Offsets.AutoCommit.Enable = false
 7
 8// RIGHT: Manual commit after successful processing
 9func (h *groupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
10    for msg := range claim.Messages() {
11        if err := h.handlers[msg.Topic](ctx, msg.Topic, msg.Key, msg.Value); err != nil {
12            continue // retry — offset does not advance
13        }
14        session.MarkMessage(msg, "")
15        session.Commit() // offset advances only after success
16    }
17    return nil
18}
19
20// Consequence: if the process crashes after publish but before commit
21// → Kafka redelivers on restart
22// → The handler must be idempotent (already handled by event_id in Article 10)

With manual commit, the contract becomes at-least-once: a message may be processed more than once during recovery, but is never lost — the right trade-off for notifications that must not be missed.


12.10 Dead Letter Queue: Implementation

The DLQ handles permanently-failing messages without blocking the consumer. The code below is a dedicated producer that sends failed messages to the DLQ topic complete with error metadata.

go
 1// internal/event/kafka/dlq_producer.go
 2
 3package kafka
 4
 5import (
 6    "context"
 7    "fmt"
 8    "time"
 9
10    "github.com/IBM/sarama"
11)
12
13const DLQTopic = "santekno.notifications.dlq"
14
15type DLQProducer struct {
16    producer sarama.SyncProducer
17}
18
19func NewDLQProducer(brokers []string) (*DLQProducer, error) {
20    config := sarama.NewConfig()
21    config.Producer.Return.Successes = true
22    producer, err := sarama.NewSyncProducer(brokers, config)
23    if err != nil {
24        return nil, fmt.Errorf("dlqProducer: %w", err)
25    }
26    return &DLQProducer{producer: producer}, nil
27}
28
29func (p *DLQProducer) SendToDLQ(ctx context.Context, originalTopic string, key, value []byte, err error) {
30    _, _, sendErr := p.producer.SendMessage(&sarama.ProducerMessage{
31        Topic: DLQTopic,
32        Key:   sarama.ByteEncoder(key),
33        Value: sarama.ByteEncoder(value),
34        Headers: []sarama.RecordHeader{
35            {Key: []byte("original-topic"), Value: []byte(originalTopic)},
36            {Key: []byte("error"),          Value: []byte(err.Error())},
37            {Key: []byte("failed-at"),      Value: []byte(time.Now().UTC().Format(time.RFC3339))},
38        },
39    })
40    if sendErr != nil {
41        fmt.Printf("DLQ send failed: %v\n", sendErr)
42    }
43}

Including original-topic, error, and failed-at in the DLQ message headers gives the ops team complete context for diagnosis — they can know what failed, why, and when without guessing.


12.11 Partition Key Strategy

The partition key determines the ordering and distribution of messages across partitions. The guide below summarizes the key strategy per topic type in Santekno Shop.

text
 1The partition key determines which partition receives a message.
 2Messages with the same key → same partition → ordered.
 3
 4Santekno Shop partition key strategy:
 5
 6santekno.orders.*:
 7  Key: order_id
 8  Benefit: all events for one order in the same partition
 9           The consumer reads events in order per order
10
11santekno.payments.*:
12  Key: order_id (not payment_id)
13  Benefit: payment events and order events correlate via order_id
14
15santekno.notifications.*:
16  Key: user_id
17  Benefit: all notifications for one user in the same partition
18           Ordering guarantee per user
19
20Implementation in the outbox worker:
21  extractKey(payload) → look for "order_id" first, then "user_id"

The decision to use order_id (not payment_id) as the key for the payment topic is an important detail: it keeps the order and payment events for the same transaction correlated in the same partition.


12.12 Monitoring: Consumer Lag

Consumer lag is the most important health metric for an event-driven system. The code below computes the total lag per topic by contacting the Kafka admin API.

go
 1// pkg/kafka/lag_monitor.go
 2
 3package kafka
 4
 5import (
 6    "context"
 7    "fmt"
 8    "time"
 9
10    "github.com/IBM/sarama"
11)
12
13type LagMonitor struct {
14    brokers []string
15    groupID string
16}
17
18func NewLagMonitor(brokers []string, groupID string) *LagMonitor {
19    return &LagMonitor{brokers: brokers, groupID: groupID}
20}
21
22// CheckLag: return the total consumer lag for all topics
23func (m *LagMonitor) CheckLag(ctx context.Context) (map[string]int64, error) {
24    client, err := sarama.NewClient(m.brokers, sarama.NewConfig())
25    if err != nil {
26        return nil, fmt.Errorf("lagMonitor.CheckLag: %w", err)
27    }
28    defer client.Close()
29
30    admin, _ := sarama.NewClusterAdminFromClient(client)
31    defer admin.Close()
32
33    // Get committed offsets per topic/partition
34    groups, _ := admin.ListConsumerGroupOffsets(m.groupID, nil)
35
36    lags := make(map[string]int64)
37    for topic, partitions := range groups.Blocks {
38        for partition, block := range partitions {
39            latest, _ := client.GetOffset(topic, partition, sarama.OffsetNewest)
40            lag := latest - block.Offset
41            if lag < 0 { lag = 0 }
42            lags[topic] += lag
43        }
44    }
45    return lags, nil
46}

A lag that keeps growing means the consumer can’t keep up with the rate of event production — a signal to add instances or optimize the handler before customer notifications fall hours behind.


12.13 Retry Policy for Failed Events in the Outbox

Events that fail to publish need to be retried with a limit and backoff. The SQL below extends the outbox table with retry columns and a polling query that respects backoff.

sql
 1-- internal/event/outbox/retry.go
 2-- If the publisher fails, the event stays in the outbox (published = false)
 3-- The worker will retry on the next poll (every 5 seconds)
 4-- For events that keep failing (e.g., Kafka down > 1 hour):
 5-- Add retry_count and last_error to the outbox_events table
 6
 7ALTER TABLE outbox_events
 8    ADD COLUMN retry_count INT NOT NULL DEFAULT 0,
 9    ADD COLUMN last_error  TEXT,
10    ADD COLUMN last_retry_at TIMESTAMPTZ;
11
12-- Query for processOnce with max retry:
13SELECT id, topic, payload, event_type
14FROM outbox_events
15WHERE published = false
16  AND retry_count < 10        -- max 10 retries
17  AND (last_retry_at IS NULL OR last_retry_at < NOW() - INTERVAL '30 seconds')
18ORDER BY created_at ASC
19LIMIT 100
20FOR UPDATE SKIP LOCKED;
21
22-- Update after a failed publish:
23UPDATE outbox_events
24SET retry_count = retry_count + 1,
25    last_error = $2,
26    last_retry_at = NOW()
27WHERE id = $1;

The retry_count < 10 limit prevents a poison event from being retried forever, while the last_retry_at filter applies backoff so a Kafka that’s down isn’t bombarded with requests every 5 seconds.


12.14 Kafka Topic Configuration

Production topics need the right retention and replication settings. The commands below set 7-day retention, minimum in-sync replicas, and a message size limit.

bash
 1# Production topic configuration
 2# Retention: 7 days (default)
 3# Cleanup policy: delete (not compact — these are events, not a log)
 4
 5kafka-configs --bootstrap-server localhost:9092 \
 6    --entity-type topics \
 7    --entity-name santekno.orders.placed \
 8    --alter \
 9    --add-config \
10    retention.ms=604800000,\  # 7 days
11    min.insync.replicas=2,\   # production: needs 2 replicas
12    max.message.bytes=1048576  # 1MB max message
13
14# Verify
15kafka-configs --bootstrap-server localhost:9092 \
16    --entity-type topics \
17    --entity-name santekno.orders.placed \
18    --describe

The min.insync.replicas=2 setting works together with RequiredAcks=WaitForAll on the producer: both must be configured consistently for the durability guarantee to actually hold, not just on paper.


12.15 Integration Test: Outbox Worker

The integration test verifies the worker actually publishes and marks events published in the database. The code below tests a single processing cycle with a mock publisher that captures events.

go
 1// internal/event/outbox/worker_integration_test.go
 2//go:build integration
 3
 4func TestOutboxWorker_PublishesToKafka(t *testing.T) {
 5    pool := setupDB(t)
 6    ctx  := context.Background()
 7
 8    // Insert a test outbox event
 9    _, err := pool.Exec(ctx,
10        `INSERT INTO outbox_events (event_type, topic, payload)
11         VALUES ($1, $2, $3)`,
12        "order.placed",
13        "santekno.orders.placed",
14        `{"order_id":"test-123","user_id":"user-456","total_amount_idr":500000}`,
15    )
16    require.NoError(t, err)
17
18    // Setup mock publisher to capture published events
19    published := make(chan event.Event, 10)
20    mockPub := &capturePublisher{ch: published}
21
22    worker := outbox.NewWorker(pool, mockPub)
23
24    // Run one processing cycle
25    err = worker.ProcessOnce(ctx) // expose processOnce for testing
26    require.NoError(t, err)
27
28    // Verify the event was published
29    select {
30    case evt := <-published:
31        assert.Equal(t, "santekno.orders.placed", evt.Topic)
32        assert.Contains(t, string(evt.Payload.(json.RawMessage)), "test-123")
33    case <-time.After(2 * time.Second):
34        t.Fatal("event not published")
35    }
36
37    // Verify the event marked as published in DB
38    var isPublished bool
39    pool.QueryRow(ctx, "SELECT published FROM outbox_events WHERE event_type = 'order.placed'").Scan(&isPublished)
40    assert.True(t, isPublished)
41}
42
43type capturePublisher struct{ ch chan event.Event }
44func (p *capturePublisher) Publish(_ context.Context, e event.Event) error {
45    p.ch <- e
46    return nil
47}

This test verifies two effects at once: the event is sent to the publisher AND the row in the database is marked published = true — ensuring the worker won’t re-publish the same event in the next cycle.


12.16 Kafka UI: Development Tool

Kafka UI speeds up debugging with a visual view of topics, messages, and consumer lag. The configuration below adds Kafka UI to the development docker-compose.

yaml
 1# docker-compose.yml — Kafka UI for development
 2
 3kafka-ui:
 4  image: provectuslabs/kafka-ui:latest
 5  ports:
 6    - "8090:8080"
 7  environment:
 8    KAFKA_CLUSTERS_0_NAME: santekno-local
 9    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
10    KAFKA_CLUSTERS_0_READONLY: "false"
11  depends_on:
12    - kafka

The features available through this UI sum up the day-to-day debugging flow, from inspecting messages to monitoring lag. The list below details their uses.

text
1Useful Kafka UI features:
2  → Topics: view partitions, retention, messages
3  → Messages: browse messages per topic/partition/offset
4  → Consumer Groups: view real-time lag
5  → Produce: test-publish a message manually
6  → Schema Registry: (optional, for Avro)
7
8Access: http://localhost:8090

With Kafka UI, developers can verify message payloads and consumer lag visually — far faster than typing kafka-console-consumer commands over and over while debugging.


12.17 Event Schema Evolution

An event schema will evolve over time, and its changes must be managed so they don’t break older consumers. The guide below distinguishes safe changes from those that need versioning.

text
 1BACKWARD COMPATIBLE CHANGES (safe):
 2  + Add a new optional field to the payload
 3  → Old consumers ignore the new field → OK
 4
 5FORWARD COMPATIBLE CHANGES (need versioning):
 6  - Remove a field from the payload
 7  - Change a field's type
 8  → New consumers can't read old events
 9
10Strategy for Santekno Shop:
11  All payload fields: an "add-only" policy
12  Never remove or rename a field from an event payload
13  Add a new field if you need to extend
14
15Versioning (Topic 9 — LLMOps):
16  event_type: "order.placed.v2" for breaking changes
17  Consumers subscribe to "order.placed.v1" AND "order.placed.v2"
18  After all consumers are updated: deprecate v1

The “add-only” policy is a simple rule that avoids 90% of compatibility problems: as long as fields are only added and never removed or retyped, old consumers keep working without changes.


12.18 Kafka Best Practices

A number of practices separate a resilient Kafka integration from a fragile one. The DO and DON’T list below summarizes the lessons we’ve applied throughout this article.

text
 1DO:
 2✓ Use an idempotent producer (config.Producer.Idempotent = true)
 3✓ Set RequiredAcks = WaitForAll for durability
 4✓ Manual commit only after processing succeeds
 5✓ Use FOR UPDATE SKIP LOCKED in outbox polling
 6✓ Design consumer handlers to be idempotent
 7✓ Set meaningful partition keys (order_id, user_id)
 8✓ Monitor consumer lag via Prometheus/Grafana
 9✓ Set the retention period according to business need
10
11DON'T:
12✗ Auto-commit (risk: message loss on crash)
13✗ A partition key with too low cardinality (e.g., status)
14   → All PENDING orders go to the same partition → hot partition
15✗ Block the Kafka consumer goroutine with expensive operations
16✗ Retry without backoff → overwhelm Kafka
17✗ Trust at-most-once delivery → always design for at-least-once

One trap that’s often missed is in the DON’T list: choosing a low-cardinality partition key like status makes all PENDING orders pile up in one partition — a hot partition that negates Kafka’s parallelism benefit.


12.19 Metrics for Kafka

Prometheus metrics give quantitative visibility into the pipeline’s health. The code below defines counters, gauges, and alert rules for the outbox and consumer lag.

go
 1// Prometheus metrics for Kafka integration
 2
 3var (
 4    outboxPublished = prometheus.NewCounterVec(
 5        prometheus.CounterOpts{Name: "outbox_events_published_total"},
 6        []string{"topic"},
 7    )
 8    outboxFailed = prometheus.NewCounterVec(
 9        prometheus.CounterOpts{Name: "outbox_events_failed_total"},
10        []string{"topic", "error"},
11    )
12    outboxPendingGauge = prometheus.NewGaugeFunc(
13        prometheus.GaugeOpts{Name: "outbox_events_pending"},
14        func() float64 {
15            var count float64
16            db.QueryRow(context.Background(),
17                "SELECT COUNT(*) FROM outbox_events WHERE published = false").Scan(&count)
18            return count
19        },
20    )
21    consumerLag = prometheus.NewGaugeVec(
22        prometheus.GaugeOpts{Name: "kafka_consumer_lag"},
23        []string{"topic", "consumer_group"},
24    )
25)
26
27// Alert rules (Prometheus):
28// ALERT OutboxBacklog IF outbox_events_pending > 1000 FOR 5m
29// ALERT ConsumerLagHigh IF kafka_consumer_lag > 500 FOR 10m

The outbox_events_pending gauge, which should always stay close to zero, is the best early alarm: if the number piles up, the worker is stuck or Kafka is having trouble — long before customers feel the impact.


12.20 Summary: Kafka Integration Complete

The Santekno Shop event flow is now fully functional end-to-end:

PlaceOrder → DB transaction (INSERT order + INSERT outbox_events) → Outbox Worker (poll every 5s, FOR UPDATE SKIP LOCKED) → Kafka Publisher (idempotent producer, RequiredAcks=All) → Kafka Topics → Consumer Group (manual commit, at-least-once) → Notification Handler (idempotent via event_id).

Outbox Worker: FOR UPDATE SKIP LOCKED for multiple instances, retry with exponential backoff, metrics for monitoring.

Consumer Group: manual commit, DLQ for permanent failures, graceful shutdown.

Partition strategy: order_id as the key → ordering guarantee per order, so the customer journey from placed to shipped always arrives in order.

With the Kafka integration complete, the entire event infrastructure is now connected. Next we optimize performance by adding a Redis caching layer per service to reduce the database load.

Related Articles

💬 Comments