Golang Kafka Integration: Outbox Worker, Producer, Consumer Group E-Commerce
A complete Kafka integration for Santekno Shop. An outbox worker that publishes to Kafka, producer setup with sarama, and a consumer group for notifications — an end-to-end event flow.
Golang Kafka integration with the outbox pattern is the backbone of Santekno Shop’s event-driven architecture that we haven’t yet completed. In Part 2, the outbox worker was still a stub — events were written to the outbox_events table but never actually published to Kafka. In this article we close that loop: a Kafka producer that publishes events from the outbox, a consumer group for the notification service, and an end-to-end test for the entire event flow.
We start from topic setup, a publisher with an idempotent producer, an outbox worker based on FOR UPDATE SKIP LOCKED, all the way to the partition key strategy for preserving event ordering.
12.01 Kafka Topics Setup
The first step is creating all the topics Santekno Shop needs. The script below creates the order, payment, and DLQ topics with 3 partitions each.
1# Setup Kafka topics for Santekno Shop
2# Run after kafka is ready in docker-compose
3
4KAFKA_BROKER=localhost:9092
5
6# Create all topics with 3 partitions, replication factor 1 (dev)
7topics=(
8 "santekno.orders.placed"
9 "santekno.orders.confirmed"
10 "santekno.orders.cancelled"
11 "santekno.orders.shipped"
12 "santekno.payments.completed"
13 "santekno.payments.failed"
14 "santekno.notifications.dlq"
15)
16
17for topic in "${topics[@]}"; do
18 kafka-topics --create \
19 --bootstrap-server $KAFKA_BROKER \
20 --topic "$topic" \
21 --partitions 3 \
22 --replication-factor 1 \
23 --if-not-exists
24 echo "Created topic: $topic"
25done
26
27# Verify
28kafka-topics --list --bootstrap-server $KAFKA_BROKER | grep santeknoChoosing 3 partitions per topic isn’t a random number: it sets the maximum number of consumer instances that can process in parallel, so plan it according to your target throughput from the start.
12.02 Kafka Publisher: Interface and Implementation
The publisher wraps sarama’s SyncProducer with a configuration that prioritizes durability and exactly-once per partition. The code below shows the full implementation plus a NoOpPublisher for development without Kafka.
1// internal/event/kafka/publisher.go
2
3package kafka
4
5import (
6 "context"
7 "encoding/json"
8 "fmt"
9
10 "github.com/IBM/sarama"
11
12 "github.com/santekno/santekno-shop/internal/event"
13)
14
15type Publisher struct {
16 producer sarama.SyncProducer
17}
18
19func NewPublisher(brokers []string) (*Publisher, error) {
20 config := sarama.NewConfig()
21 config.Producer.Return.Successes = true
22 config.Producer.Return.Errors = true
23 config.Producer.RequiredAcks = sarama.WaitForAll // acks from all ISR
24 config.Producer.Retry.Max = 3
25 config.Producer.Compression = sarama.CompressionSnappy
26
27 // Idempotent producer: exactly-once per partition
28 config.Producer.Idempotent = true
29 config.Net.MaxOpenRequests = 1 // required for idempotent
30
31 producer, err := sarama.NewSyncProducer(brokers, config)
32 if err != nil {
33 return nil, fmt.Errorf("kafkaPublisher.New: %w", err)
34 }
35
36 return &Publisher{producer: producer}, nil
37}
38
39func (p *Publisher) Publish(ctx context.Context, e event.Event) error {
40 payload, err := json.Marshal(e.Payload)
41 if err != nil {
42 return fmt.Errorf("kafkaPublisher.Publish: marshal: %w", err)
43 }
44
45 msg := &sarama.ProducerMessage{
46 Topic: e.Topic,
47 Value: sarama.ByteEncoder(payload),
48 }
49
50 // Use event key for partition routing (same key = same partition = ordered)
51 if e.Key != "" {
52 msg.Key = sarama.StringEncoder(e.Key)
53 }
54
55 _, _, err = p.producer.SendMessage(msg)
56 if err != nil {
57 return fmt.Errorf("kafkaPublisher.Publish [%s]: %w", e.Topic, err)
58 }
59
60 return nil
61}
62
63func (p *Publisher) Close() error {
64 return p.producer.Close()
65}
66
67// NoOpPublisher: for development without Kafka running
68type NoOpPublisher struct{}
69
70func (p *NoOpPublisher) Publish(_ context.Context, e event.Event) error {
71 fmt.Printf("[EVENT] topic=%s key=%s\n", e.Topic, e.Key)
72 return nil
73}The combination of Idempotent = true and RequiredAcks = WaitForAll is the key: the first prevents duplicates caused by network retries, the second guarantees data isn’t lost until every ISR replica has confirmed.
12.03 Outbox Worker: Full Implementation
The outbox worker is the bridge between PostgreSQL (ACID) and Kafka (eventually consistent). The code below polls unpublished events, sends them to Kafka, then marks them published.
1// internal/event/outbox/worker.go
2
3package outbox
4
5import (
6 "context"
7 "encoding/json"
8 "fmt"
9 "time"
10
11 "github.com/jackc/pgx/v5/pgxpool"
12 "github.com/santekno/santekno-shop/internal/event"
13)
14
15type Worker struct {
16 db *pgxpool.Pool
17 publisher event.Publisher
18 batchSize int
19 interval time.Duration
20 stopCh chan struct{}
21}
22
23func NewWorker(db *pgxpool.Pool, publisher event.Publisher) *Worker {
24 return &Worker{
25 db: db,
26 publisher: publisher,
27 batchSize: 100,
28 interval: 5 * time.Second,
29 stopCh: make(chan struct{}),
30 }
31}
32
33func (w *Worker) Run(ctx context.Context) {
34 ticker := time.NewTicker(w.interval)
35 defer ticker.Stop()
36
37 fmt.Println("Outbox worker started")
38 for {
39 select {
40 case <-ctx.Done():
41 fmt.Println("Outbox worker stopped (context)")
42 return
43 case <-w.stopCh:
44 fmt.Println("Outbox worker stopped (signal)")
45 return
46 case <-ticker.C:
47 if err := w.processOnce(ctx); err != nil {
48 fmt.Printf("outbox worker error: %v\n", err)
49 }
50 }
51 }
52}
53
54func (w *Worker) Stop() {
55 close(w.stopCh)
56}
57
58// processOnce: take a batch of unpublished events, publish to Kafka
59func (w *Worker) processOnce(ctx context.Context) error {
60 // Fetch unpublished events (SKIP LOCKED for multiple worker instances)
61 rows, err := w.db.Query(ctx,
62 `SELECT id, topic, payload, event_type
63 FROM outbox_events
64 WHERE published = false
65 ORDER BY created_at ASC
66 LIMIT $1
67 FOR UPDATE SKIP LOCKED`, // prevent duplicate processing
68 w.batchSize,
69 )
70 if err != nil {
71 return fmt.Errorf("outbox.processOnce: fetch: %w", err)
72 }
73 defer rows.Close()
74
75 type outboxRow struct {
76 ID string
77 Topic string
78 Payload json.RawMessage
79 EventType string
80 }
81
82 var events []outboxRow
83 for rows.Next() {
84 var row outboxRow
85 if err := rows.Scan(&row.ID, &row.Topic, &row.Payload, &row.EventType); err != nil {
86 return fmt.Errorf("outbox.processOnce: scan: %w", err)
87 }
88 events = append(events, row)
89 }
90 rows.Close()
91
92 if len(events) == 0 {
93 return nil // nothing to do
94 }
95
96 // Publish each event
97 published := 0
98 for _, e := range events {
99 if err := w.publisher.Publish(ctx, event.Event{
100 Topic: e.Topic,
101 Key: extractKey(e.Payload), // extract order_id or user_id
102 Payload: e.Payload,
103 }); err != nil {
104 // Log but don't stop — try the next event
105 fmt.Printf("outbox: publish failed [%s]: %v\n", e.EventType, err)
106 continue
107 }
108
109 // Mark as published
110 if _, err := w.db.Exec(ctx,
111 `UPDATE outbox_events
112 SET published = true, published_at = NOW()
113 WHERE id = $1`,
114 e.ID,
115 ); err != nil {
116 fmt.Printf("outbox: mark published failed [%s]: %v\n", e.ID, err)
117 }
118 published++
119 }
120
121 if published > 0 {
122 fmt.Printf("outbox: published %d/%d events\n", published, len(events))
123 }
124 return nil
125}
126
127// extractKey: extract the partition key from the JSON payload
128func extractKey(payload json.RawMessage) string {
129 var m map[string]interface{}
130 if err := json.Unmarshal(payload, &m); err != nil {
131 return ""
132 }
133 for _, key := range []string{"order_id", "user_id", "payment_id"} {
134 if val, ok := m[key].(string); ok && val != "" {
135 return val
136 }
137 }
138 return ""
139}The FOR UPDATE SKIP LOCKED clause in the fetch query is what makes the worker safe to run multi-instance: each worker locks the rows it takes and skips rows already locked by another worker, so no event gets published twice.
12.04 Consumer Group: Order Event Consumer
This generic consumer group maps topics to handlers and applies manual commit. The code below is a reusable version of the consumer that can be used across services.
1// internal/event/kafka/consumer_group.go
2
3package kafka
4
5import (
6 "context"
7 "fmt"
8 "log"
9
10 "github.com/IBM/sarama"
11)
12
13type MessageHandler func(ctx context.Context, topic string, key, value []byte) error
14
15type ConsumerGroup struct {
16 client sarama.ConsumerGroup
17 topics []string
18 handlers map[string]MessageHandler
19}
20
21func NewConsumerGroup(brokers []string, groupID string, handlers map[string]MessageHandler) (*ConsumerGroup, error) {
22 config := sarama.NewConfig()
23 config.Consumer.Group.Rebalance.GroupStrategies = []sarama.BalanceStrategy{
24 sarama.NewBalanceStrategyRoundRobin(),
25 }
26 config.Consumer.Offsets.Initial = sarama.OffsetNewest
27 config.Consumer.Offsets.AutoCommit.Enable = false // manual commit only
28
29 client, err := sarama.NewConsumerGroup(brokers, groupID, config)
30 if err != nil {
31 return nil, fmt.Errorf("kafka.NewConsumerGroup: %w", err)
32 }
33
34 topics := make([]string, 0, len(handlers))
35 for topic := range handlers {
36 topics = append(topics, topic)
37 }
38
39 return &ConsumerGroup{
40 client: client,
41 topics: topics,
42 handlers: handlers,
43 }, nil
44}
45
46func (cg *ConsumerGroup) Run(ctx context.Context) error {
47 handler := &groupHandler{handlers: cg.handlers}
48 for {
49 if err := cg.client.Consume(ctx, cg.topics, handler); err != nil {
50 log.Printf("consumer group error: %v", err)
51 }
52 if ctx.Err() != nil {
53 return ctx.Err()
54 }
55 }
56}
57
58func (cg *ConsumerGroup) Close() error {
59 return cg.client.Close()
60}
61
62type groupHandler struct {
63 handlers map[string]MessageHandler
64}
65
66func (h *groupHandler) Setup(sarama.ConsumerGroupSession) error { return nil }
67func (h *groupHandler) Cleanup(sarama.ConsumerGroupSession) error { return nil }
68
69func (h *groupHandler) ConsumeClaim(
70 session sarama.ConsumerGroupSession,
71 claim sarama.ConsumerGroupClaim,
72) error {
73 for msg := range claim.Messages() {
74 handler, ok := h.handlers[msg.Topic]
75 if !ok {
76 session.MarkMessage(msg, "")
77 continue
78 }
79
80 ctx := context.Background()
81 if err := handler(ctx, msg.Topic, msg.Key, msg.Value); err != nil {
82 // Don't commit — Kafka will redeliver
83 log.Printf("handler error [%s]: %v", msg.Topic, err)
84 continue
85 }
86
87 // Commit only on success
88 session.MarkMessage(msg, "")
89 session.Commit()
90 }
91 return nil
92}The MessageHandler type as a function signature makes this consumer agnostic to the domain — any service can register its own handler without changing the core consumer group code.
12.05 Wiring: Main Server with Kafka
All the components are assembled in the main server, with Kafka optional via a config flag. The code below unifies the publisher, outbox worker, consumer, and HTTP server in a single lifecycle.
1// cmd/server/main.go — updated with Kafka
2
3func main() {
4 cfg, _ := config.Load()
5 ctx, cancel := context.WithCancel(context.Background())
6 defer cancel()
7
8 // Database
9 pool, _ := pgxpool.New(ctx, cfg.Database.URL)
10 defer pool.Close()
11 rdb := redis.NewClient(&redis.Options{Addr: cfg.Redis.URL})
12 defer rdb.Close()
13
14 // Kafka Publisher
15 var publisher event.Publisher
16 if cfg.Kafka.Enabled {
17 kafkaPublisher, err := kafkaPkg.NewPublisher(cfg.Kafka.Brokers)
18 if err != nil {
19 log.Fatalf("kafka publisher: %v", err)
20 }
21 defer kafkaPublisher.Close()
22 publisher = kafkaPublisher
23 } else {
24 publisher = &event.NoOpPublisher{} // development without Kafka
25 }
26
27 // Build container (all usecases)
28 container := BuildContainer(cfg, pool, rdb, publisher)
29
30 // Outbox Worker
31 outboxWorker := outbox.NewWorker(pool, publisher)
32 go outboxWorker.Run(ctx)
33
34 // Kafka Consumer (notification service)
35 if cfg.Kafka.Enabled {
36 notifHandlers := buildNotificationHandlers(pool)
37 consumer, _ := kafkaPkg.NewConsumerGroup(
38 cfg.Kafka.Brokers,
39 "santekno-notification",
40 notifHandlers,
41 )
42 defer consumer.Close()
43 go consumer.Run(ctx)
44 }
45
46 // HTTP server
47 e := SetupRouter(container, cfg)
48 go e.Start(":" + cfg.App.Port)
49
50 // Graceful shutdown
51 quit := make(chan os.Signal, 1)
52 signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
53 <-quit
54
55 outboxWorker.Stop()
56 shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
57 defer shutdownCancel()
58 e.Shutdown(shutdownCtx)
59 cancel() // stop consumer goroutines
60}The cfg.Kafka.Enabled flag lets Kafka be disabled in development and replaced by NoOpPublisher — developers can run the application without having to spin up a local Kafka cluster.
12.06 Config: Kafka Settings
The Kafka configuration is read from environment variables with sensible defaults. The code below defines KafkaConfig and how it’s loaded.
1// config/config.go
2
3type KafkaConfig struct {
4 Enabled bool
5 Brokers []string
6 GroupID string
7}
8
9func Load() (*Config, error) {
10 return &Config{
11 // ...
12 Kafka: KafkaConfig{
13 Enabled: getEnvBool("KAFKA_ENABLED", true),
14 Brokers: strings.Split(getEnv("KAFKA_BROKERS", "localhost:9092"), ","),
15 GroupID: getEnv("KAFKA_GROUP_ID", "santekno-notification"),
16 },
17 }, nil
18}
19
20// .env.example
21// KAFKA_ENABLED=true
22// KAFKA_BROKERS=localhost:9092
23// KAFKA_GROUP_ID=santekno-notificationKeeping the brokers and group ID in environment variables keeps the application portable across environments — from a developer’s laptop to a production cluster — without changing code.
12.07 End-to-End Test: PlaceOrder → Kafka Event
The end-to-end test proves the entire flow works: an order created via API, published by the worker, and received in Kafka. The code below runs that scenario with a direct sarama consumer.
1// tests/e2e/order_flow_test.go
2//go:build e2e
3
4package e2e_test
5
6import (
7 "context"
8 "encoding/json"
9 "testing"
10 "time"
11
12 "github.com/IBM/sarama"
13 "github.com/stretchr/testify/assert"
14 "github.com/stretchr/testify/require"
15)
16
17// TestOrderFlow_PlaceOrder_PublishesKafkaEvent:
18// 1. Place an order via HTTP API
19// 2. Wait for the outbox worker to publish
20// 3. Verify the event is received in Kafka
21func TestOrderFlow_PlaceOrder_PublishesKafkaEvent(t *testing.T) {
22 ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
23 defer cancel()
24
25 // Setup Kafka consumer to capture events
26 config := sarama.NewConfig()
27 config.Consumer.Offsets.Initial = sarama.OffsetNewest
28 consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, config)
29 require.NoError(t, err)
30 defer consumer.Close()
31
32 partConsumer, err := consumer.ConsumePartition("santekno.orders.placed", 0, sarama.OffsetNewest)
33 require.NoError(t, err)
34 defer partConsumer.Close()
35
36 // Place order via API (using test helper)
37 orderID := placeTestOrder(t)
38
39 // Wait for the outbox worker to publish (max 15 seconds)
40 select {
41 case msg := <-partConsumer.Messages():
42 var payload map[string]interface{}
43 require.NoError(t, json.Unmarshal(msg.Value, &payload))
44
45 assert.Equal(t, orderID, payload["order_id"])
46 assert.NotEmpty(t, payload["user_id"])
47 assert.NotEmpty(t, payload["total_amount_idr"])
48 t.Logf("Event received: order_id=%s", orderID)
49
50 case err := <-partConsumer.Errors():
51 t.Fatalf("consumer error: %v", err)
52
53 case <-ctx.Done():
54 t.Fatal("timeout: event not received within 15 seconds")
55 }
56}
57
58func placeTestOrder(t *testing.T) string {
59 t.Helper()
60 // Login, place order via HTTP, return order_id
61 // (simplified — actual HTTP calls to the test server)
62 return "test-order-id-xxx"
63}This test verifies the system’s most important contract: an order created by a customer really does produce an event in Kafka within seconds — something an isolated unit test can’t prove.
12.08 FOR UPDATE SKIP LOCKED: Concurrency Safety
To understand why FOR UPDATE SKIP LOCKED is crucial, let’s compare the scenario with and without it. The diagram below illustrates how two workers could process the same event without the lock.
1PROBLEM: multiple outbox worker instances running
2
3Without SKIP LOCKED:
4 Worker A: SELECT WHERE published=false LIMIT 100
5 Worker B: SELECT WHERE published=false LIMIT 100
6 → Both take the same events!
7 → Duplicate publish to Kafka
8
9WITH FOR UPDATE SKIP LOCKED:
10 Worker A: SELECT ... FOR UPDATE SKIP LOCKED
11 → Lock rows, skip those already locked by another worker
12 Worker B: SELECT ... FOR UPDATE SKIP LOCKED
13 → Skip rows locked by Worker A
14 → Take rows not yet locked
15
16Result: each event is processed by only one worker
17Note: the Kafka consumer must still be idempotent (Kafka itself can redeliver)Note the last remark: SKIP LOCKED prevents duplicates on the producer side, but Kafka can still redeliver on the consumer side — that’s why idempotency in the handler (via the event_id from Article 10) remains mandatory.
12.09 Kafka Offset Management
The offset commit strategy determines whether the system loses messages on a crash. The code below contrasts the dangerous auto-commit with the safe manual commit.
1// The correct offset commit strategy for the notification service
2
3// WRONG: AutoCommit enabled
4// → Message committed before it's processed → message loss on crash
5
6config.Consumer.Offsets.AutoCommit.Enable = false
7
8// RIGHT: Manual commit after successful processing
9func (h *groupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
10 for msg := range claim.Messages() {
11 if err := h.handlers[msg.Topic](ctx, msg.Topic, msg.Key, msg.Value); err != nil {
12 continue // retry — offset does not advance
13 }
14 session.MarkMessage(msg, "")
15 session.Commit() // offset advances only after success
16 }
17 return nil
18}
19
20// Consequence: if the process crashes after publish but before commit
21// → Kafka redelivers on restart
22// → The handler must be idempotent (already handled by event_id in Article 10)With manual commit, the contract becomes at-least-once: a message may be processed more than once during recovery, but is never lost — the right trade-off for notifications that must not be missed.
12.10 Dead Letter Queue: Implementation
The DLQ handles permanently-failing messages without blocking the consumer. The code below is a dedicated producer that sends failed messages to the DLQ topic complete with error metadata.
1// internal/event/kafka/dlq_producer.go
2
3package kafka
4
5import (
6 "context"
7 "fmt"
8 "time"
9
10 "github.com/IBM/sarama"
11)
12
13const DLQTopic = "santekno.notifications.dlq"
14
15type DLQProducer struct {
16 producer sarama.SyncProducer
17}
18
19func NewDLQProducer(brokers []string) (*DLQProducer, error) {
20 config := sarama.NewConfig()
21 config.Producer.Return.Successes = true
22 producer, err := sarama.NewSyncProducer(brokers, config)
23 if err != nil {
24 return nil, fmt.Errorf("dlqProducer: %w", err)
25 }
26 return &DLQProducer{producer: producer}, nil
27}
28
29func (p *DLQProducer) SendToDLQ(ctx context.Context, originalTopic string, key, value []byte, err error) {
30 _, _, sendErr := p.producer.SendMessage(&sarama.ProducerMessage{
31 Topic: DLQTopic,
32 Key: sarama.ByteEncoder(key),
33 Value: sarama.ByteEncoder(value),
34 Headers: []sarama.RecordHeader{
35 {Key: []byte("original-topic"), Value: []byte(originalTopic)},
36 {Key: []byte("error"), Value: []byte(err.Error())},
37 {Key: []byte("failed-at"), Value: []byte(time.Now().UTC().Format(time.RFC3339))},
38 },
39 })
40 if sendErr != nil {
41 fmt.Printf("DLQ send failed: %v\n", sendErr)
42 }
43}Including original-topic, error, and failed-at in the DLQ message headers gives the ops team complete context for diagnosis — they can know what failed, why, and when without guessing.
12.11 Partition Key Strategy
The partition key determines the ordering and distribution of messages across partitions. The guide below summarizes the key strategy per topic type in Santekno Shop.
1The partition key determines which partition receives a message.
2Messages with the same key → same partition → ordered.
3
4Santekno Shop partition key strategy:
5
6santekno.orders.*:
7 Key: order_id
8 Benefit: all events for one order in the same partition
9 The consumer reads events in order per order
10
11santekno.payments.*:
12 Key: order_id (not payment_id)
13 Benefit: payment events and order events correlate via order_id
14
15santekno.notifications.*:
16 Key: user_id
17 Benefit: all notifications for one user in the same partition
18 Ordering guarantee per user
19
20Implementation in the outbox worker:
21 extractKey(payload) → look for "order_id" first, then "user_id"The decision to use order_id (not payment_id) as the key for the payment topic is an important detail: it keeps the order and payment events for the same transaction correlated in the same partition.
12.12 Monitoring: Consumer Lag
Consumer lag is the most important health metric for an event-driven system. The code below computes the total lag per topic by contacting the Kafka admin API.
1// pkg/kafka/lag_monitor.go
2
3package kafka
4
5import (
6 "context"
7 "fmt"
8 "time"
9
10 "github.com/IBM/sarama"
11)
12
13type LagMonitor struct {
14 brokers []string
15 groupID string
16}
17
18func NewLagMonitor(brokers []string, groupID string) *LagMonitor {
19 return &LagMonitor{brokers: brokers, groupID: groupID}
20}
21
22// CheckLag: return the total consumer lag for all topics
23func (m *LagMonitor) CheckLag(ctx context.Context) (map[string]int64, error) {
24 client, err := sarama.NewClient(m.brokers, sarama.NewConfig())
25 if err != nil {
26 return nil, fmt.Errorf("lagMonitor.CheckLag: %w", err)
27 }
28 defer client.Close()
29
30 admin, _ := sarama.NewClusterAdminFromClient(client)
31 defer admin.Close()
32
33 // Get committed offsets per topic/partition
34 groups, _ := admin.ListConsumerGroupOffsets(m.groupID, nil)
35
36 lags := make(map[string]int64)
37 for topic, partitions := range groups.Blocks {
38 for partition, block := range partitions {
39 latest, _ := client.GetOffset(topic, partition, sarama.OffsetNewest)
40 lag := latest - block.Offset
41 if lag < 0 { lag = 0 }
42 lags[topic] += lag
43 }
44 }
45 return lags, nil
46}A lag that keeps growing means the consumer can’t keep up with the rate of event production — a signal to add instances or optimize the handler before customer notifications fall hours behind.
12.13 Retry Policy for Failed Events in the Outbox
Events that fail to publish need to be retried with a limit and backoff. The SQL below extends the outbox table with retry columns and a polling query that respects backoff.
1-- internal/event/outbox/retry.go
2-- If the publisher fails, the event stays in the outbox (published = false)
3-- The worker will retry on the next poll (every 5 seconds)
4-- For events that keep failing (e.g., Kafka down > 1 hour):
5-- Add retry_count and last_error to the outbox_events table
6
7ALTER TABLE outbox_events
8 ADD COLUMN retry_count INT NOT NULL DEFAULT 0,
9 ADD COLUMN last_error TEXT,
10 ADD COLUMN last_retry_at TIMESTAMPTZ;
11
12-- Query for processOnce with max retry:
13SELECT id, topic, payload, event_type
14FROM outbox_events
15WHERE published = false
16 AND retry_count < 10 -- max 10 retries
17 AND (last_retry_at IS NULL OR last_retry_at < NOW() - INTERVAL '30 seconds')
18ORDER BY created_at ASC
19LIMIT 100
20FOR UPDATE SKIP LOCKED;
21
22-- Update after a failed publish:
23UPDATE outbox_events
24SET retry_count = retry_count + 1,
25 last_error = $2,
26 last_retry_at = NOW()
27WHERE id = $1;The retry_count < 10 limit prevents a poison event from being retried forever, while the last_retry_at filter applies backoff so a Kafka that’s down isn’t bombarded with requests every 5 seconds.
12.14 Kafka Topic Configuration
Production topics need the right retention and replication settings. The commands below set 7-day retention, minimum in-sync replicas, and a message size limit.
1# Production topic configuration
2# Retention: 7 days (default)
3# Cleanup policy: delete (not compact — these are events, not a log)
4
5kafka-configs --bootstrap-server localhost:9092 \
6 --entity-type topics \
7 --entity-name santekno.orders.placed \
8 --alter \
9 --add-config \
10 retention.ms=604800000,\ # 7 days
11 min.insync.replicas=2,\ # production: needs 2 replicas
12 max.message.bytes=1048576 # 1MB max message
13
14# Verify
15kafka-configs --bootstrap-server localhost:9092 \
16 --entity-type topics \
17 --entity-name santekno.orders.placed \
18 --describeThe min.insync.replicas=2 setting works together with RequiredAcks=WaitForAll on the producer: both must be configured consistently for the durability guarantee to actually hold, not just on paper.
12.15 Integration Test: Outbox Worker
The integration test verifies the worker actually publishes and marks events published in the database. The code below tests a single processing cycle with a mock publisher that captures events.
1// internal/event/outbox/worker_integration_test.go
2//go:build integration
3
4func TestOutboxWorker_PublishesToKafka(t *testing.T) {
5 pool := setupDB(t)
6 ctx := context.Background()
7
8 // Insert a test outbox event
9 _, err := pool.Exec(ctx,
10 `INSERT INTO outbox_events (event_type, topic, payload)
11 VALUES ($1, $2, $3)`,
12 "order.placed",
13 "santekno.orders.placed",
14 `{"order_id":"test-123","user_id":"user-456","total_amount_idr":500000}`,
15 )
16 require.NoError(t, err)
17
18 // Setup mock publisher to capture published events
19 published := make(chan event.Event, 10)
20 mockPub := &capturePublisher{ch: published}
21
22 worker := outbox.NewWorker(pool, mockPub)
23
24 // Run one processing cycle
25 err = worker.ProcessOnce(ctx) // expose processOnce for testing
26 require.NoError(t, err)
27
28 // Verify the event was published
29 select {
30 case evt := <-published:
31 assert.Equal(t, "santekno.orders.placed", evt.Topic)
32 assert.Contains(t, string(evt.Payload.(json.RawMessage)), "test-123")
33 case <-time.After(2 * time.Second):
34 t.Fatal("event not published")
35 }
36
37 // Verify the event marked as published in DB
38 var isPublished bool
39 pool.QueryRow(ctx, "SELECT published FROM outbox_events WHERE event_type = 'order.placed'").Scan(&isPublished)
40 assert.True(t, isPublished)
41}
42
43type capturePublisher struct{ ch chan event.Event }
44func (p *capturePublisher) Publish(_ context.Context, e event.Event) error {
45 p.ch <- e
46 return nil
47}This test verifies two effects at once: the event is sent to the publisher AND the row in the database is marked published = true — ensuring the worker won’t re-publish the same event in the next cycle.
12.16 Kafka UI: Development Tool
Kafka UI speeds up debugging with a visual view of topics, messages, and consumer lag. The configuration below adds Kafka UI to the development docker-compose.
1# docker-compose.yml — Kafka UI for development
2
3kafka-ui:
4 image: provectuslabs/kafka-ui:latest
5 ports:
6 - "8090:8080"
7 environment:
8 KAFKA_CLUSTERS_0_NAME: santekno-local
9 KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
10 KAFKA_CLUSTERS_0_READONLY: "false"
11 depends_on:
12 - kafkaThe features available through this UI sum up the day-to-day debugging flow, from inspecting messages to monitoring lag. The list below details their uses.
1Useful Kafka UI features:
2 → Topics: view partitions, retention, messages
3 → Messages: browse messages per topic/partition/offset
4 → Consumer Groups: view real-time lag
5 → Produce: test-publish a message manually
6 → Schema Registry: (optional, for Avro)
7
8Access: http://localhost:8090With Kafka UI, developers can verify message payloads and consumer lag visually — far faster than typing kafka-console-consumer commands over and over while debugging.
12.17 Event Schema Evolution
An event schema will evolve over time, and its changes must be managed so they don’t break older consumers. The guide below distinguishes safe changes from those that need versioning.
1BACKWARD COMPATIBLE CHANGES (safe):
2 + Add a new optional field to the payload
3 → Old consumers ignore the new field → OK
4
5FORWARD COMPATIBLE CHANGES (need versioning):
6 - Remove a field from the payload
7 - Change a field's type
8 → New consumers can't read old events
9
10Strategy for Santekno Shop:
11 All payload fields: an "add-only" policy
12 Never remove or rename a field from an event payload
13 Add a new field if you need to extend
14
15Versioning (Topic 9 — LLMOps):
16 event_type: "order.placed.v2" for breaking changes
17 Consumers subscribe to "order.placed.v1" AND "order.placed.v2"
18 After all consumers are updated: deprecate v1The “add-only” policy is a simple rule that avoids 90% of compatibility problems: as long as fields are only added and never removed or retyped, old consumers keep working without changes.
12.18 Kafka Best Practices
A number of practices separate a resilient Kafka integration from a fragile one. The DO and DON’T list below summarizes the lessons we’ve applied throughout this article.
1DO:
2✓ Use an idempotent producer (config.Producer.Idempotent = true)
3✓ Set RequiredAcks = WaitForAll for durability
4✓ Manual commit only after processing succeeds
5✓ Use FOR UPDATE SKIP LOCKED in outbox polling
6✓ Design consumer handlers to be idempotent
7✓ Set meaningful partition keys (order_id, user_id)
8✓ Monitor consumer lag via Prometheus/Grafana
9✓ Set the retention period according to business need
10
11DON'T:
12✗ Auto-commit (risk: message loss on crash)
13✗ A partition key with too low cardinality (e.g., status)
14 → All PENDING orders go to the same partition → hot partition
15✗ Block the Kafka consumer goroutine with expensive operations
16✗ Retry without backoff → overwhelm Kafka
17✗ Trust at-most-once delivery → always design for at-least-onceOne trap that’s often missed is in the DON’T list: choosing a low-cardinality partition key like status makes all PENDING orders pile up in one partition — a hot partition that negates Kafka’s parallelism benefit.
12.19 Metrics for Kafka
Prometheus metrics give quantitative visibility into the pipeline’s health. The code below defines counters, gauges, and alert rules for the outbox and consumer lag.
1// Prometheus metrics for Kafka integration
2
3var (
4 outboxPublished = prometheus.NewCounterVec(
5 prometheus.CounterOpts{Name: "outbox_events_published_total"},
6 []string{"topic"},
7 )
8 outboxFailed = prometheus.NewCounterVec(
9 prometheus.CounterOpts{Name: "outbox_events_failed_total"},
10 []string{"topic", "error"},
11 )
12 outboxPendingGauge = prometheus.NewGaugeFunc(
13 prometheus.GaugeOpts{Name: "outbox_events_pending"},
14 func() float64 {
15 var count float64
16 db.QueryRow(context.Background(),
17 "SELECT COUNT(*) FROM outbox_events WHERE published = false").Scan(&count)
18 return count
19 },
20 )
21 consumerLag = prometheus.NewGaugeVec(
22 prometheus.GaugeOpts{Name: "kafka_consumer_lag"},
23 []string{"topic", "consumer_group"},
24 )
25)
26
27// Alert rules (Prometheus):
28// ALERT OutboxBacklog IF outbox_events_pending > 1000 FOR 5m
29// ALERT ConsumerLagHigh IF kafka_consumer_lag > 500 FOR 10mThe outbox_events_pending gauge, which should always stay close to zero, is the best early alarm: if the number piles up, the worker is stuck or Kafka is having trouble — long before customers feel the impact.
12.20 Summary: Kafka Integration Complete
The Santekno Shop event flow is now fully functional end-to-end:
PlaceOrder → DB transaction (INSERT order + INSERT outbox_events) → Outbox Worker (poll every 5s, FOR UPDATE SKIP LOCKED) → Kafka Publisher (idempotent producer, RequiredAcks=All) → Kafka Topics → Consumer Group (manual commit, at-least-once) → Notification Handler (idempotent via event_id).
Outbox Worker: FOR UPDATE SKIP LOCKED for multiple instances, retry with exponential backoff, metrics for monitoring.
Consumer Group: manual commit, DLQ for permanent failures, graceful shutdown.
Partition strategy: order_id as the key → ordering guarantee per order, so the customer journey from placed to shipped always arrives in order.
With the Kafka integration complete, the entire event infrastructure is now connected. Next we optimize performance by adding a Redis caching layer per service to reduce the database load.