01
Platform
Orchestration, probes, and the failure of a control plane.
7Every platform incident p99 has published — what broke, what the operator was told, and the fix that held.
02
Incidents
7- A 4-minute brownout became a 29-minute outage. Every layer retried correctly.retries · retry-budget · amplification · backoff · load-sheddingmedium
- The nightly job had run for three years. One Sunday in March it didn't, and nothing failed.cron · dst · timezones · scheduler · missed-runs · monitoringmedium
- One partition stopped moving at 11:02. Nothing alerted. Support did, at 14:41.partitions · consumer-lag · head-of-line-blocking · dead-letter-queue · retriesmedium
- A generated config file turns one gradual database rollout into repeating edge outagesconfig-propagation · rust · clickhouse · fail-static · blast-radiushard
- No deploy, no traffic change — every internal call fails at exactly 00:00:07tls · mtls · certificates · expiry · observabilityeasy
- One customer's 50,000 req/min spreads across 20 instances. Only shared state sees the whole stream.redis · rate-limiting · distributed-systems · concurrency · slahard
- Traffic spikes. Pods start dying. Healthy pods, killed by their own cluster.kubernetes · cascading-failure · load-sheddinghard
→