The S3 Command That Took Out us-east-1
How a maintenance command with one wrong input removed too much capacity from two foundational S3 subsystems in us-east-1, and why recovery was gated by cold-start paths that had not run at large-region scale in years.
Before S3 could return or place an object, it had to know where that object belonged. The index subsystem tracked object metadata and location; the placement subsystem decided where new objects should go. Both needed enough live capacity to absorb routine maintenance. Remove too much capacity too quickly, and a regional service that normally hides server churn no longer has a safe path for reads, writes, deletes, lists, or new object placement.
On the afternoon of February 28, 2017, an engineer was investigating a billing issue in us-east-1. The debugging step was familiar: take a small set of servers out of the billing subsystem to isolate the problem, then put them back. The command took a scope argument — a number specifying how many servers to target. The engineer entered a value much larger than they intended. The tooling did not pause. It did not check whether the removal would push a subsystem below its minimum required capacity. It accepted the command and ran it. The engineer had meant to remove a handful of servers from a billing-related subsystem. The input removed a much larger set of servers supporting the index and placement subsystems. Within minutes, S3 could no longer service GET, LIST, PUT, or DELETE requests in us-east-1. The failure also reached AWS services that depended on regional S3 storage: EC2 new instance launches, EBS volumes that needed S3 snapshots, Lambda, and the S3 console. AWS also hit a communication problem. The Service Health Dashboard remained visible, but the administration console used to update individual service status depended on S3, so AWS had to use the AWS Twitter feed and dashboard banner text until that path came back.
Restoring service meant fully restarting both affected subsystems. The procedures existed and worked in development, but AWS had not completely restarted the index or placement subsystems in its larger regions for years. S3 had grown, and the safety checks required to validate metadata integrity took longer than expected at us-east-1 scale. The index activated enough capacity to serve GET, LIST, and DELETE at 20:26 UTC and fully recovered at 21:18. PUT still needed placement. Placement finished at 21:54, four hours and seventeen minutes after the command ran, and dependent AWS services then began clearing their backlogs. One wrong input triggered the outage; the load-bearing design flaw was a maintenance tool that could remove capacity faster than the subsystem could safely lose it.
From the first signal to all-clear in 4h 17m.
An S3 team member runs an established playbook command intended to remove a small number of servers from a billing-related subsystem. One input is entered incorrectly, and the command removes a much larger set of servers than intended.
The removed servers support the index subsystem, which tracks object metadata and location, and the placement subsystem, which allocates storage for new writes. Both subsystems require full restarts before S3 can serve requests normally.
S3 cannot service GET, LIST, PUT, and DELETE requests in us-east-1 while the affected subsystems restart. AWS services that rely on S3 in the region, including EC2 launches, EBS snapshot-backed volumes, Lambda, and the S3 console, are also impacted.
Engineers determine that the index and placement subsystems must be restarted. AWS has not fully restarted these subsystems in its larger regions for years.
Restart proceeds slower than expected because metadata safety checks and capacity activation take longer at us-east-1 scale.
AWS can update individual service status again after relying on the AWS Twitter feed and banner text. The SHD administration console had depended on S3.
The index subsystem activates enough capacity to begin serving GET, LIST, and DELETE requests. PUT requests still require the placement subsystem to recover.
The index subsystem fully recovers. The placement subsystem continues restoring the capacity needed for new object placement and PUT requests.
The placement subsystem finishes recovery and S3 returns to normal operation in us-east-1, four hours and seventeen minutes after the command ran.