Library — Failure Modes — Failure Modes

/

sort

newsletter

Get the next incident.

One production failure case study every week. Read the pattern before it shows up in your own system.

28 results

id

incident

org

date

duration

severity

tags

The Overheated AWS ZonetrendingA thermal event in one US-EAST-1 data center impaired EC2 instances and EBS volumes in use1-az4, disrupting workloads that depended on resources pinned to the affected Availability Zone.

us-east-1use1-az4thermal-event

The Encryption Path Under Slack MessagesSlack EKM customers experienced message sending, channel loading, workflow, notification, DM, and file-operation issues after elevated encryption-key request load turned a security dependency into an availability bottleneck.

enterprise-key-managementkmscaching

The DNSSEC Failure That Made .de Look FakeIncorrect DNSSEC signatures for Germany's .de top-level domain caused validating resolvers to reject .de answers, leading Cloudflare to temporarily bypass DNSSEC validation for the zone.

denicde-dnssecdns

The Search Layer That Slowed GitHubA concentrated wave of anonymous scraping traffic saturated the load-balancing tier in front of GitHub Search, causing timeouts across issues, pull requests, repositories, Actions, packages, and Dependabot alerts.

searchscrapingload-balancing

The Silent Merge Queue Corruption That Hit 658 GitHub ReposA half-gated feature flag let an unreleased merge-base path escape into squash merge groups. Over a 4h38m impact window, GitHub's merge queue produced valid-looking commits that silently reverted prior work across 658 repositories and 2,092 pull requests.

merge-queuefeature-flagsquash-merge

The WAF Killswitch That Crashed the Older ProxyA global WAF testing-tool killswitch exposed an FL1 proxy bug, returning HTTP 500s for sites using the older proxy and Managed Ruleset.

The Bot File That Crashed Cloudflare's ProxyA ClickHouse permissions change duplicated Bot Management feature rows, producing an oversized file that crashed Cloudflare proxy traffic paths.

configwafbot-management

Azure Front Door config crashes edge sitesIncompatible Azure Front Door customer metadata exposed a data-plane bug, crashing edge sites and causing connection timeouts and DNS errors globally.

Microsoft Azure

The DynamoDB DNS Race That Emptied US-EAST-1A DynamoDB DNS automation race emptied the US-EAST-1 regional endpoint, then cascaded into EC2 launch, NLB, Lambda, and console failures.

dnsautomationdynamodb

Service Control quota update rejects APIsA quota policy update with blank fields crashed Google Service Control globally, causing 503s across Google Cloud, Workspace, and Security Operations APIs.

configquotaspanner

1 / 3