Short-form technical writing — one engineering idea at a time. The long-form versions live on the blog; the conversations happen on LinkedIn .
·Distributed Systems·1 min read
Your microservices are losing events. Quietly.
If a service writes to its database and then publishes to Kafka, there is a crash window where the commit succeeds and the event never leaves the building. It happens rarely — maybe once a month — which is exactly why nobody notices until the data has drifted. The fix costs one table. Write the event to an outbox row in the same transaction as the business change, let a relay publish it, and make consumers idempotent. Atomicity becomes the database's problem, and databases are very good at that problem.
In multi-tenant systems, 'default tenant' is where bugs go to hide
Code review heuristic that has caught real incidents: search the codebase for tenant fallbacks. Every `getTenantOrDefault()` is a future cross-tenant leak wearing a convenience costume. A request that reaches the data layer without a resolved tenant should throw — loudly, immediately, in staging. Fail closed. The exception you see today is the data leak you don't see next quarter.
Review AI-generated code like a PR from a brilliant hire who's never seen production
Brilliant: stop reviewing syntax, it's clean. Never seen production: check every transaction boundary, every retry, every assumption about ordering and time. AI code fails exactly where systems fail — at the boundaries — and the polish of everything around the bug is what lulls reviewers past it. The engineers getting real leverage from AI aren't the ones who prompt best. They're the ones who review best.
'The query uses an index' is not a performance claim
EXPLAIN ANALYZE showed an index scan — and 'Rows Removed by Filter: 214,893'. The index was used, and then 99.98% of what it read was thrown away. Selectivity is the goal, not index usage. One composite index whose column order matched the actual query shape (equality columns first, then the sort column) deleted the sort node entirely and cut the endpoint's p95 by more than half. Read your plans with production-sized parameters or don't bother reading them.
The most reliable component in our platform is a while-loop
The outbox relay: poll a table, publish to Kafka, mark rows, back off when idle. No framework, no cleverness, ~200 lines. It has survived broker outages, deploy storms, and a database failover, because every failure mode is 'try again in a moment.' There's a lesson in there about where to spend your complexity budget. Spend it on the invariants — exactly-once effect, ordering per aggregate — and make everything else boring enough to be obviously correct.
We stopped treating search as a query feature and everything got easier
Search over millions of records kept fighting us as long as it was 'flexible SQL filtering.' The unlock was conceptual: search is a read model. Denormalize documents that contain everything a result needs, project them from domain events, compile user queries through a whitelist so the safe path is the only path. Eventual consistency became something we could design for and say out loud — 'results update within seconds' — instead of a bug report generator.