The Leap-Day Arithmetic Error That Froze Every New Azure VM for 27 Hours

At midnight on February 29, 2012, every new and reincarnated Azure VM hit a date arithmetic error in the Guest Agent. Adding one year to February 29 produced the invalid date February 29, 2013, and transfer certificate creation failed immediately. A transfer certificate is the credential a VM generates during initialization so it can register with the platform controller. Transfer certificate creation was the first step in Guest Agent initialization — required before any VM could register with the Fabric Controller. The failure hit new deployments, scale-outs, OS updates, and service-healing reincarnations simultaneously.

Windows Azure organized physical servers into clusters of roughly 1,000, each managed by a redundant Fabric Controller. The Fabric Controller was responsible for deployments, scaling, health monitoring, and automatic service healing. When the Guest Agent terminated silently before connecting, the Host Agent waited through three consecutive 25-minute timeouts. After those timeouts, it classified the server as having a probable hardware fault and moved it to Human Investigate state.

Before enough servers reached the Human Investigate threshold, service healing moved VMs to other servers. On those new servers, the same Guest Agent bug reproduced, cascading more servers into Human Investigate state. Critical alarms fired at 01:15 UTC, stopped the rollout, and paged Fabric Controller developers. Developers identified the leap-day bug at 02:38 UTC. At 02:55 UTC, Microsoft disabled service management in all clusters worldwide. This blocked customer deployments, scale-outs, and OS updates, preventing additional VMs from hitting the failing certificate path.

The Guest Agent had no retry or graceful-degradation path on certificate failure. When the date calculation produced an invalid date, the agent terminated immediately. There was no retry, no partial initialization mode, and no typed error surfaced to the Host Agent. The 75-minute timeout sequence created a window in which certificate failures accumulated across rolling clusters. Each affected VM took three 25-minute cycles to move its server to HI state, while clusters kept placing new VMs on additional servers, each beginning its own countdown.

Microsoft validated the Guest Agent fix in a test cluster at 06:50 UTC and confirmed it in one production cluster. At 10:11 UTC, Microsoft pushed the fix to all clusters. Seven clusters had just started their platform rollout when the bug struck. Most of their servers ran the old Host Agent and Guest Agent combination; some had already moved to the new version. For these clusters, Microsoft chose a blast update. The blast pushed the older Host Agent to all servers simultaneously, bypassing the standard multi-hour staged-domain rollout. The blast package included a networking plugin written for the newer Host Agent, incompatible with the older one it was paired with. The single-server pre-push test confirmed VMs started but did not verify network connectivity. At 10:47 UTC, the package disconnected every VM in all seven clusters from the network, including previously healthy ones. Access Control Service and Service Bus were deployed inside those seven clusters. When every VM disconnected, any application depending on those services lost them regardless of its own Compute footprint.

Disabling service management worldwide at 02:55 UTC was a containment decision with a clear rationale. Customer scale-out, deployment, and OS update operations all created new VMs, and every new VM would attempt the same failing certificate path. The tradeoff was visible — customers lost the ability to manage their applications — but the freeze preserved already-running multi-instance services that still had healthy capacity. The seven clusters were largely operational by 16:00 UTC, but the final healthy status did not arrive until 10:15 UTC on March 1. Servers left in corrupted intermediate states by repeated transitions required per-server manual restoration that automated tooling could not complete.

Microsoft's postmortem framed remediation across multiple layers. Time-boundary tests and typed error classification would catch date bugs and surface Guest Agent failures without the 75-minute timeout. Finer-grained service management controls would let individual aspects be disabled without a global freeze. Microsoft committed to investment in tooling to speed recovery from intermediate states.