The Transactional Outbox, Properly

Every microservices codebase eventually contains this method, and it is always wrong:

@Transactional
public void approveClaim(ClaimId id) {
    claim.approve();
    repository.save(claim);
    kafka.send("claim-events", new ClaimApproved(id));  // the bug
}

If the send happens inside the transaction, you announce events for state that may roll back. Move it after the commit, and a crash between commit and send loses the event forever. The database and the broker are two systems; no ordering of two writes makes them one.

This failure is quiet and rare, which is what makes it dangerous. Your notification service misses one approval a month. Your analytics drift by a fraction of a percent. Nobody files a bug — they just slowly stop trusting the data.

The pattern

Write the event to an outbox table in the same transaction as the business change. Atomicity is now the database's problem, and databases are good at that problem:

@Transactional
public void approveClaim(ClaimId id) {
    claim.approve();
    repository.save(claim);
    outbox.save(OutboxRecord.of(new ClaimApproved(id)));  // same transaction
}

A relay — a small, boring loop — polls the outbox and publishes to Kafka:

SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;   -- multiple relay instances without stepping on each other

FOR UPDATE SKIP LOCKED is the detail most write-ups omit: it lets you run more than one relay instance for availability without double-publishing on every poll (you'll still get occasional duplicates on crash — that's the next section's job).

At-least-once means duplicates. Plan for them.

The relay can crash after publishing but before marking the row. Kafka can redeliver on consumer rebalance. Your delivery guarantee is at-least-once, so consumers must be idempotent:

@KafkaListener(topics = "claim-events")
@Transactional
public void on(ClaimApproved event) {
    if (processedEvents.existsById(event.eventId())) return;  // duplicate — skip
    handle(event);
    processedEvents.save(new ProcessedEvent(event.eventId()));  // same transaction
}

The idempotency check and the side effect commit together, so a crash mid-handler redelivers and retries cleanly.

"Why not Kafka's exactly-once semantics?" EOS is real, but it covers Kafka-to-Kafka pipelines. The moment your consumer writes to Postgres or calls an API, you need idempotency anyway. Consumer-side idempotency keys work with any delivery guarantee and are honest about the failure modes. I stopped reaching for EOS entirely.

The parts nobody diagrams

• Ordering. Use the aggregate id as the Kafka message key so all events for one entity land on one partition, in order. Cross-aggregate ordering doesn't exist; design so you don't need it.
• Poison messages. A message that fails deterministically will block its partition forever under naive retry. Retry a few times, then move it to a dead-letter topic with the exception attached, and alert. DLQ traffic is a bug report, not garbage.
• Outbox growth. Published rows must be deleted or archived on a schedule, or your fastest- growing table is the one with no business value.
• Relay tuning. Poll fast, back off when empty. We measured: 50ms polling with exponential backoff to 1s idle gave sub-100ms publish latency at negligible database load.

Trade-offs, stated plainly

You've added a table, a relay process, and a processed-events table per consumer. In exchange: no lost events, no phantom events, replayable history, and consumers that survive redelivery. For systems where events drive money or state, that trade is not close.

The runnable version of everything above — including a chaos script that kills processes mid-flight so you can watch it recover — is on GitHub as kafka-outbox.