Problem
The dual-write problem: a service updates its database and publishes an event to Kafka, but those are two systems and two failure domains. Publish before commit and you announce state that may roll back; publish after commit and a crash in between silently loses the event. Downstream services drift out of sync and nobody notices until reconciliation.
Business Context
Most explanations of the outbox pattern are diagrams; this is a runnable system. It exists as the
reference I wanted when introducing reliable eventing into a production microservices platform —
something a team can clone, run with one docker compose up, kill processes mid-flight, and watch
recover without losing an event.
Architecture
Business transaction ──┐
Outbox INSERT ─────────┴─ same DB transaction
↓
Relay (polls outbox) → Kafka topic → Idempotent consumer
→ Dead-letter topic (poison messages)
Events are written to an outbox table in the same transaction as the business change — atomicity comes free from the database. A relay polls the outbox and publishes to Kafka with an ordering key. Consumers record processed event ids and skip duplicates; messages that repeatedly fail go to a dead-letter topic for inspection and replay.
Technology
Java 17, Spring Boot, Spring Kafka, PostgreSQL, Docker Compose. The demo harness includes a chaos script that kills the relay and consumers at random intervals to demonstrate recovery.
Challenges
The interesting decision was idempotency versus Kafka's exactly-once semantics. EOS transactions are real but confined to Kafka-to-Kafka pipelines; the moment a consumer touches an external database, you need idempotency anyway. Consumer-side idempotency keys — a processed-events table checked in the consumer's transaction — cost one lookup per message and work with any delivery guarantee. Simpler, portable, and honest about the failure modes.
Tuning the relay was the second lesson: polling interval trades latency against database load, and the answer ("poll fast, back off when empty") only emerged from measuring.
Solution
A minimal, production-shaped implementation of at-least-once delivery with exactly-once effect: no lost events, duplicates safely ignored, poison messages quarantined instead of blocking the partition.
Results
- Zero lost events by construction, demonstrated under randomized process kills.
- Used as the pattern template for reliable eventing in a production platform.
- A CDC-based relay variant (Debezium) is on the roadmap for comparison.
Lessons Learned
Embrace at-least-once and design idempotency in from the start — it is simpler and more portable than chasing end-to-end exactly-once. Reference implementations teach better than diagrams.