The Regex That Burned Every Cloudflare CPU
How a regex with one unbounded subpattern, deployed globally with no canary stage, burned every CPU core in Cloudflare's edge network — and why a global kill switch built years earlier turned a long outage into a 27-minute one.
Every Cloudflare edge server had to make the same promise on every request: inspect quickly, then forward. The Web Application Firewall was part of that promise. It matched HTTP requests against managed rules before traffic reached customer origins. As long as each rule consumed bounded CPU, the WAF behaved like a thin filter. If one rule took unbounded time on common request shapes, inspection stopped being a filter and became the thing that consumed the proxy.
On the afternoon of July 2, 2019, an engineer merged a pull request adding new managed rules aimed at a class of JavaScript-based attacks. The pipeline did what it always did. CI built the rule set, ran the test suite, and approved it for deployment. Six minutes after the build, the rule was on every edge server in the network. Unlike Cloudflare's general software releases, which first reached an internal dogfood point of presence before going global, WAF rules were treated as data and pushed everywhere at once. One of the new rules contained a regex with an unbounded inner subpattern: .*(?:.*=.*). On most strings it matched or rejected quickly. On a class of live HTTP bodies, it entered catastrophic backtracking — the engine explored exponentially many ways the pattern might match before concluding it could not. The WAF's regex engine had no per-evaluation time limit. Every core that processed HTTP traffic ran the rule on the same request shapes and stalled at nearly 100% CPU at the same time. HTTP and HTTPS traffic across the network collapsed within minutes. The blast radius reached Cloudflare's own controls. The dashboard, the API, and the status page all served through the same edge proxy that was now too busy to forward anything. Engineers correlating the CPU spike with the WAF deployment could see the cause, but the usual path to disable the rule depended on the system the rule had just taken down.
What ended the incident was a piece of tooling built years earlier for exactly this category of failure: a global terminate mechanism that disabled the WAF on every edge server without going through the regular configuration path. The team executed it at 14:07 UTC. Two minutes later, traffic and CPU were back to normal. The WAF stayed off while engineers isolated the bad rule, then came back online with the rest of the rule set. Twenty-seven minutes of global edge outage, caused by a single unbounded subpattern in a regex — and a deploy procedure that put rule changes on a fast path that code changes were not allowed to use.
From the first signal to all-clear in 27m.
An engineer merges a pull request adding new managed WAF rules targeting JavaScript-based attacks. One of the rules contains a regular expression with an unbounded subpattern.
Cloudflare's build system compiles the rule set and runs the test suite against synthetic inputs. The tests pass — checking correctness, not CPU cost — and the rule is approved for production deployment.
The standard procedure for WAF rule changes is to deploy them to the entire network at once. The rule lands on every edge server within seconds.
Synthetic checks against the WAF begin failing. CPU usage on cores handling HTTP and HTTPS traffic climbs toward 100% as the regex enters catastrophic backtracking on live request bodies.
HTTP and HTTPS error rates spike globally — an 82% traffic drop. The Cloudflare dashboard, API, and status page — all served through the same edge network — also become unreachable.
Engineers correlate the CPU saturation with the just-deployed WAF rule set. Standard rollback paths are slow because the dashboard used to manage rules is itself unreachable through the edge.
Engineers execute the global terminate — a kill switch built years earlier specifically to disable the WAF without depending on the normal deploy pipeline. The WAF stops evaluating rules on every edge server.
CPU usage drops back to baseline almost immediately. HTTP and HTTPS traffic returns to expected levels worldwide. Total customer impact: 27 minutes.
After isolating the bad rule and testing the rest, engineers re-enable the WAF globally with the offending rule removed.