The library. Every incident, structured.

A growing archive of public postmortems, broken down into a consistent shape: what broke, why it cascaded, and what to take from it. New incidents added regularly.

29+
incidents
11+
years
13
organizations
/
sort

newsletter

Get the next incident.

One production failure case study every week. Read the pattern before it shows up in your own system.

1 result · filtered
topic: data-loss
id
incident
org
date
duration
severity
tags
FM-006
The `rm -rf` That Erased GitLab's Production DatabasetrendingA sysadmin accidentally deleted GitLab.com's production PostgreSQL database. The normal backups were broken or unsuitable, so GitLab restored from a six-hour-old LVM snapshot.
GitLab
2017-01-31
18h 30m
SEV-1
databasepostgresqlbackup