CrowdStrike has hired two outside security firms to review the Falcon functionality that sparked a global IT outage last month - but it may not have an awful lot to find, because CrowdStrike has identified the simple mistake that caused the meltdown.

News of the external review emerged in a root causes analysis [PDF] published on Tuesday by the infosec vendor.

As we learned from CrowdStrike's earlier post-incident write-up of the flawed Falcon update, which bricked millions of Windows machines worldwide, the problem began back in February.

That was when the developer added to Falcon, its threat-detection suite, the ability to spot and block the novel exploitation of named pipes and other Windows interprocess communication (IPC) mechanisms; seeing such attacks occur is a strong indication that the box has been compromised, which is good to stop and flag up.

That new detection functionality went through the usual development and testing before CrowdStrike pushed it as a new "template type" to customers' Falcon installations in sensor version 7.11.

These template types are as the name suggests: Templates. They are generalized routines for each picking up different types of potentially bad activity on a system. For Falcon to use them to detect specific threats, so-called template instances are defined by CrowdStrike that customize the template routines to identify particular forms of exploitation and other bad stuff.

Since March, CrowdStrike has remotely pushed from its cloud to Falcon deployments a handful of template instances that made use of the IPC template type to detect specific threats. These updates are stored in a channel file numbered 291. Falcon would fetch an updated channel 291 file, containing the new instances, and have its Content Interpreter parse the information.

That info would tell Falcon how to use the template type to perform the desired detection. The root causes analysis provided a deeper look at what went wrong next:

Then, as CrowdStrike also previously explained, two further IPC-related Template Instances were automatically deployed to Falcon users on July 19. One of these used a non-wildcard matching criterion for the 21st input. This resulted in a new version of the Channel File that required Falcon sensors to inspect the 21 inputs - but another piece of software called the Content Interpreter expected only 20 values.

"Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash," the security shop explained in the root cause analysis.

CrowdStrike has coded a fix to ensure that mismatches of the number of inputs validated versus number of actual inputs doesn't happen again. It's a patch for the Sensor Content Compiler - this is the function that validates the number of inputs provided by the template type - and it went into production July 27.

CrowdStrike also wrote that it has added runtime input array bounds checks to the Content Interpreter for Rapid Response updates, to ensure the size of the input array matches the number of expected inputs. These fixes are currently being backported to all Windows sensor versions 7.11 and above with a sensor software hotfix. The release will be generally available by August 9.

Additionally, the chastened security vendor is doing more tests - including some that test non-wildcard matching criteria for each field across all template types, and new checks to ensure that flawed files aren't pushed to Falcon customers in the future.

Further, as CrowdStrike had noted in its earlier analysis, every Template Instance will henceforth be deployed to customers in a staged rollout, rather than being pushed to all users all at once.

It's worth noting that the company is being sued by investors for not originally using this type of phased approach in sending updates to customers.

"Looking ahead, CrowdStrike is focused on using the lessons learned from this incident to better serve our customers," a spokesperson declared. "CrowdStrike remains steadfast in our mission to protect customers and stop breaches."

But not so steadfast that it’s naming the partners it hired to review its code.

Those reviews have commenced, and are focused on the code and processes that led to the July 19 fiasco.

"We are not providing information on the vendors who are doing work for us beyond what is referenced in the RCA," the CrowdStrike spokesperson told The Register. ®

https://www.theregister.com//2024/08/07/crowdstrike_full_incident_root_cause_analysis/