Cloudflare had a rough morning today (November 18). Its new postmortem lays out how a “small” internal database change managed to disrupt huge portions of the internet.
The outage began at 11:20 UTC. Sites behind Cloudflare suddenly started throwing 5xx errors. Dashboards stalled. Turnstile CAPTCHA failed. Workers KV choked. Cloudflare Access logins stopped working. For a company that powers so much of the web, this was a major hit.
Cloudflare confirmed the incident wasn’t an attack. The real culprit was a permissions update on a ClickHouse database. That small change caused a Bot Management feature file to unexpectedly double in size.
That file gets pushed across Cloudflare’s global network every few minutes. When machines received the oversized version, a strict internal limit inside the core proxy triggered a panic. That panic produced the massive 5xx storm customers saw.
Things became more confusing because the file regenerated every five minutes. Sometimes it was correct. Sometimes it wasn’t. That pattern briefly made the team think they might be under a fluctuating high volume attack.
Once engineers realized every node had begun producing the bad version, the trail led back to the malformed Bot Management file. Cloudflare stopped propagation, pushed a known good file, restarted core proxy services, and traffic began recovering at 14:30 UTC.
Workers KV, Cloudflare Access, Turnstile, and parts of the dashboard were all affected. Cloudflare says some email security signals degraded, but customer impact there was limited.
Cloudflare now describes this as its worst outage since 2019. The company is planning changes like hardening configuration ingestion, adding more kill switches, and reducing system overload during panic conditions.
As someone who runs NERDS.xyz on Cloudflare, I watched this unfold in real time. Thankfully, NERDS.xyz didn’t seem to experience any noticeable downtime. Still, it’s a reminder that even the largest infrastructure can wobble when a tiny assumption fails.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz