The library. Every incident, structured.

A growing archive of public postmortems, broken down into a consistent shape: what broke, why it cascaded, and what to take from it. New incidents added regularly.

28+
incidents
11+
years
13
organizations
/
sort

newsletter

Get the next incident.

One production failure case study every week. Read the pattern before it shows up in your own system.

2 results · filtered
topic: failover
id
incident
org
date
duration
severity
tags
FM-027
The Runner Cache Bug That Queued Ubuntu CI JobsA backend cache misconfiguration after failover caused duplicate GitHub Actions job assignments, reducing Ubuntu-24 runner capacity for public repos.
GitHub
2025-05-28
5h
SEV-2
cischedulerfailover
FM-002
43 Seconds of Split-Brain at GitHubA 43-second network partition between GitHub's East and West Coast sites tripped automatic failover. By the time the partition healed, both coasts had taken writes and reconciling the split took most of a day.
GitHub
2018-10-21
24h 11m
SEV-1
databasemysqlreplication