The library. Every incident, structured.

A growing archive of public postmortems, broken down into a consistent shape: what broke, why it cascaded, and what to take from it. New incidents added regularly.

29+
incidents
11+
years
13
organizations
/
sort

newsletter

Get the next incident.

One production failure case study every week. Read the pattern before it shows up in your own system.

2 results · filtered
topic: operator-error
id
incident
org
date
duration
severity
tags
FM-007
The Cleanup Script That Deleted 883 Atlassian SitesA maintenance script meant to deactivate a deprecated standalone app instead permanently deleted full customer sites. 775 customers lost access to their Jira and Confluence data, and bringing them back took up to two weeks.
Atlassian
2022-04-05
14d
SEV-1
jiraconfluenceopsgenie
FM-006
The `rm -rf` That Erased GitLab's Production DatabasetrendingA sysadmin accidentally deleted GitLab.com's production PostgreSQL database. The normal backups were broken or unsuitable, so GitLab restored from a six-hour-old LVM snapshot.
GitLab
2017-01-31
18h 30m
SEV-1
databasepostgresqlbackup