~/library/FM-021
FM-021CrowdStrike2024-07-19impact ~10 daysSEV-1

Channel File 291 turned a content update into a kernel crash.

CrowdStrike deployed new Rapid Response Content for an IPC detection template. The content validator expected 21 inputs, but Windows sensor integration supplied 20, exposing an out-of-bounds read and crashing affected machines.

kernelendpoint-securityout-of-boundswindowscontent-updatecrashfalconbsod

CrowdStrike Falcon could update some detection logic without shipping a new sensor binary. Rapid Response Content traveled through channel files, and the Windows sensor interpreted that content against fields supplied by built-in template code. The safety condition was simple: the content definition and the sensor code had to agree on how many fields existed.

Channel File 291 broke that agreement in a way earlier tests did not expose. The IPC Template Type definition described 21 input fields, while the Windows sensor integration supplied 20 values to the Content Interpreter. Earlier content and tests used wildcard criteria for the 21st field, so the mismatch stayed dormant.

On July 19, 2024, CrowdStrike deployed two new IPC Template Instances. One used a non-wildcard match on the 21st field. When affected Windows sensors received the content and later processed an IPC notification, the interpreter tried to inspect data beyond the 20 supplied inputs. The result was an out-of-bounds read and a system crash.

The recovery work had two layers. Customers had to bring affected Windows systems back, while CrowdStrike changed the content toolchain: compile-time input-count validation, runtime bounds checks, broader non-wildcard test coverage, and staged deployment for Template Instances. The durable pattern is the gap between the schema and the integration. The template definition said 21 fields. The sensor supplied 20. Earlier content never touched the 21st, so the gap stayed dormant — until a content update forced the interpreter to read it.

The Content Interpreter expected only 20 values.// CrowdStrike RCA, August 2024

From the first signal to all-clear in ~10 days.

Feb 2024
IPC template type ships
CrowdStrike sensor version 7.11 introduced a new IPC Template Type for named pipes and other Windows interprocess communication activity.
Jul 19 2024
New Channel File 291 content deployed
Two new IPC Template Instances were deployed through Channel File 291. One used a non-wildcard criterion for the 21st input field.
Jul 19 2024
Sensors crash on IPC notifications
Affected Windows sensors evaluated the new content at the next IPC notification. The Content Interpreter attempted to read the missing 21st input and triggered a system crash.
Jul 19 2024
Compiler validation patch developed
CrowdStrike developed a Sensor Content Compiler patch to validate input field counts and checked other template types for the same mismatch.
Jul 25 2024
Runtime bounds checks added
Bounds checks were added to the Content Interpreter, with an additional size check to compare provided inputs with the content's expected input count.
Jul 29 2024
Most sensors back online
CrowdStrike reported that about 99% of Windows sensors were online compared with pre-incident levels, using a week-over-week comparison.

A content schema mismatch reached kernel-adjacent runtime.

The immediate cause was a mismatch between the IPC Template Type definition and the Windows sensor integration. The content system validated Template Instances as if 21 input fields would be supplied, while the sensor code supplied only 20. A new non-wildcard match on the 21st field forced the Content Interpreter to access input data that was not present.

The systemic cause was that several validation layers tested the content interface indirectly and missed the exact production shape. Earlier tests used wildcard matching in the 21st field, the validator trusted the template definition, runtime bounds checks were missing, and Rapid Response Content lacked staged rollout controls strong enough to stop a crashing update before broad distribution.

What turned detection content into fleet recovery.

01
The failing content exercised a previously unused field.
Earlier Channel File 291 versions did not use a non-wildcard criterion for the 21st input. That allowed the template mismatch to exist in production until a normal content update finally reached the bad path.
02
Validation relied on the declared schema.
The Content Validator checked new content against the Template Type definition, which said 21 inputs existed. It did not prove that the sensor integration actually provided that many inputs to the interpreter.
03
Runtime did not defend the boundary.
A kernel-mode security product must treat content as untrusted input. Without an input-count check at the interpreter boundary, a configuration mismatch could become a system crash instead of a rejected content item.
04
Deployment controls lagged content risk.
Rapid Response Content exists to move faster than full sensor releases. That speed needs canaries, acceptance checks, rollback controls, and customer scheduling because the content is interpreted by software running deep in the endpoint.

What to take from this incident.

01
Validate both sides of a generated interface.Schema validation is incomplete if it checks only the producer's declared contract. Add compile-time and runtime checks that compare generated definitions, integration code, and actual argument counts.
02
Fuzz every interpreted content handler.Interpreters should reject malformed, incomplete, oversized, and semantically strange content without crashing. Make fuzzing and non-wildcard field coverage mandatory for each handler before content can ship.
03
Give fast content the same rollout discipline as code.Configuration that changes runtime behavior can be as dangerous as executable code. Use staged rings, bake time, crash telemetry gates, and emergency rollback for content updates with privileged execution paths.

Read the original.

External Technical Root Cause Analysis - Channel File 291
crowdstrike.com
← previous
FM-020 · The DynamoDB DNS Race That Emptied US-EAST-1
next →
FM-022 · The Bot File That Crashed Cloudflare's Proxy