A generated config file turns one gradual database rollout into repeating edge outages

01Symptom

You run a global edge network. At 11:28 UTC customers start seeing 5xx pages. Your tests catch it by 11:31, and the error rate spikes, falls to baseline, and spikes again every few minutes. You recently weathered record DDoS attacks, so the pattern looks like another one. The status page, which is hosted outside your own infrastructure, also goes down.

02Constraints

• A feature file is pushed to the entire network every few minutes so new bot tactics can be deployed quickly
• A query on a ClickHouse cluster generates that file every five minutes, while the cluster is being updated node by node
• The bot module preallocates memory for 200 features; normal use is about 60
• Two proxy generations are live: the old proxy and the new proxy
• Object storage, identity proxying, and the web login challenge widget all depend on the same core proxy

03Evidence

• →The 5xx rate jumps at 11:28, recovers, jumps again, and only later fails continuously
• →A database access-control change finished deploying at 11:05, 23 minutes before the first errors
• →Every failing new-proxy worker logs `thread new_proxy_worker_thread panicked: called Result::unwrap() on an Err value`
• →Failing feature files contain well over 200 features; good files contain about 60
• →The file-building query asks for a table's column names without restricting it to one database
• →On the new proxy, bot-dependent traffic gets 5xx. On the old proxy, it keeps serving but every bot score becomes zero
• →CPU use rises because debugging and observability systems spend it on uncaught errors
• →The first hour and a half points at degraded object storage, which is adjacent infrastructure rather than the cause

→The question

Why did the 5xx rate follow the database rollout, and what should the proxy have done with an invalid feature file?

04Your prediction