Nearly three weeks after a botched CrowdStrike update caused one of the biggest IT outages in history, the firm has published its in-depth investigation into what happened and why. CrowdStrike’s Root Cause Analysis report elaborates on the information previously shared in its preliminary Post Incident Review.
In its new post mortem report, CrowdStrike delves deeper into the root causes of the error that led Windows machines to display blue screen of death—and admits its testing process left a lot to be desired.
The firm has certainly faced a tough time in the weeks since the outage, after it was sued by investors last week. CrowdStrike and the CEO of Delta are also exchanging words after the airline blamed the security company for $500 million of losses.
What Happened
In its RCA, the firm describes how its CrowdStrike Falcon sensor “delivers AI and machine learning to protect customer systems by identifying and remediating the latest advanced threats.”
The problem that led to the outage stems from a new feature that was added to its sensor in February, “to enable visibility into possible novel attack techniques that may abuse certain Windows mechanisms.”
MORE FROMFORBES ADVISOR
This capability, which used a pre-defined a set of fields for Rapid Response Content to gather data, was developed and tested according to the firm’s “standard software development processes.”
But these do not appear to have been good enough to catch the issue.
“On March 5, 2024, following a successful stress test, the first Rapid Response Content for Channel File 291 was released to production as part of a content configuration update, with three additional Rapid Response updates deployed between April 8, 2024 and April 24, 2024,” CrowdStrike said. These “performed as expected” in production.
However, on July 19, 2024, a Rapid Response Content update was delivered to certain Windows hosts, “evolving the new capability first released in February 2024.”
The sensor expected 20 input fields, but the update provided 21 input fields. “In this instance, the mismatch resulted in an out-of-bounds memory read, causing a system crash,” CrowdStrike wrote.
This scenario with Channel File 291 is now “incapable of recurring,” CrowdStrike said, adding that what happened is now informing how it tests things going forward.
ForbesMicrosoft Confirms New Outage Was Triggered By CyberattackBy Kate O'Flaherty
What’s Next, According To The Root Cause Analysis
Based on the findings in the RCA, CrowdStrike said it will update content configuration system test procedures, including upgraded tests for template type development, with “automated tests for all existing template types.”
It’s also adding deployment layers and acceptance checks for the content configuration system.
A lot of people have complained about not having the ability to control updates. From now, CrowdStrike will provide customers additional control over the deployment of Rapid Response Content updates.
Meanwhile, it will prevent the creation of problematic Channel 291 files by implementing validation for the number of input fields.
CrowdStrike will also implement additional checks in the content validator and enhance bounds checking in the content interpreter for Rapid Response Content in Channel File 291.
Finally, it will engage “two independent third-party software security vendors” to conduct further review of the Falcon sensor code and quality control and release processes.
“Looking ahead, CrowdStrike is focused on using the lessons learned from this incident to better serve our customers,” the firm said in an emailed statement. “CrowdStrike remains steadfast in our mission to protect customers and stop breaches.”
Experts Have Their Say
The RCA is a long document—12 pages to be precise—but experts have praised CrowdStrike for being transparent about what happened on July 19. “It is positive they have been transparent and that they are taking this seriously,” says cybersecurity consultant Daniel Card.
Yet point six of the RCA, “template instances should have a staged deployment” seems to have garnered the attention of cybersecurity experts.
As a result of its RCA investigation, CrowdStrike says its content configuration system has been updated “with additional deployment layers and acceptance checks.” In other words, it will now deploy canary testing and roll things back if issues are found. At the same time, customers can now choose where and when Rapid Response Content updates like the one that caused the crash are deployed.
However, Card points out a company deploying globally to military, critical national infrastructure and governments should have had a robust test process in place in the first instance. “I'm happy to see they are going to change the design to be what it should have had in the first place.”
He points out the critical balance CrowdStrike needed to manage. “When you run this type of software, you can't be taking risks. Releasing globally without adequate testing is not a risk you want to be taking lightly as a company, but that seems to have been the design.”
The report shows that the problem wasn’t that the update was malformed—or that that the validator didn’t catch the error before it was deployed, Florian Roth, head of research at Nextron Systems, a competitor of CrowdStrike, wrote on X, formerly Twitter. “The real problem is that they didn’t have any canary system to which they deployed the update before rolling it out to millions of endpoints all at once.”
CrowdStrike has hit back at Roth’s claims. “Since our founding, we have always put customer protection at the forefront, and that continues to be our singular focus,” a CrowdStrike spokesperson wrote on an email.
“As we describe in the RCA, the parameter count mismatch evaded the multiple layers of build validation and testing in our process. The RCA also describes the process improvements and mitigation steps that CrowdStrike is deploying to ensure further enhanced resilience.”
The changes to its processes come better late than never, but the general feeling seems to be that CrowdStrike shouldn’t have been in this situation in the first place.
