Today’s incident
Order created successfully. The next request says it doesn't exist.
Databasesmediumpostgres / replication / consistency / read-after-write
Diagnose this incident Make your prediction. Then reveal the fix.
01Symptom
Your checkout API returns 201 with a new order ID, then the browser redirects to that order's page. Some customers see a 404. Refreshing a few seconds later brings the order back. The problem started after GET requests were moved to a read pool to reduce primary load, and becomes much more frequent during the nightly bulk import. Checkout success rate stays green, and there are no failed inserts.
02Constraints
- One PostgreSQL primary accepts writes; two asynchronous streaming replicas serve GET requests, selected independently for each request
- POST /orders commits before returning 201; the redirected GET uses a new transaction, with no application or CDN cache on this path
- A customer must be able to view an order immediately after creation; the order-detail latency budget is 300 ms
- The primary has capacity for order-detail reads, but moving the entire catalog and reporting workload back would overload it
- Replica replay usually trails the primary by less than 100 ms, but can fall several seconds behind during imports
03Evidence
- For one affected order, the commit completes at 14:02:10.120 and the API returns 201 at 14:02:10.125; the redirected GET returns 404 at 14:02:10.180
- At 14:02:10.200, a fresh query on the primary finds the row, while the same query on replica B returns zero rows
- Replica B's replay position is still before that transaction's commit record; it passes the record at 14:02:13.400, after which the row is visible there
- Request traces show a successful GET on replica A followed by a 404 on replica B for the same order; no DELETE statement ran
- Routing only order-detail reads to the primary eliminates the 404s during a canary run, including during the import
Your call
How can a committed order disappear between requests, what proves the cause, and how would you guarantee the immediate read without moving every read back to the primary?
Write down your diagnosis →02
Recent
Previously diagnosed.
- A 4-minute brownout became a 29-minute outage. Every layer retried correctly.retries · retry-budget · amplification · backoff · load-sheddingmedium
- The nightly job had run for three years. One Sunday in March it didn't, and nothing failed.cron · dst · timezones · scheduler · missed-runs · monitoringmedium
- One partition stopped moving at 11:02. Nothing alerted. Support did, at 14:41.partitions · consumer-lag · head-of-line-blocking · dead-letter-queue · retriesmedium
- Duplicate primary keys on one node, for about two seconds, once a monthclocks · id-generation · ntp · upsert · silent-corruptionmedium
The next incident, in your inbox
Keep your debugging instincts sharp.
One production incident a day. Symptoms, evidence, and a diagnosis to work through.