A combination of factors caused the Falcon EDR sensor to crash, resulting in the global outage affecting over 8.5 million Windows systems back in July, CrowdStrike said last week in a root cause analysis of the incident. At the same time, CrowdStrike CTO George Kurtz and president Michael Sentonas were in Las Vegas with a public mea culpa.
CrowdStrike documented in its root cause analysis that there was a mismatch between inputs validated by a Content Validator and those provided to a Content Interpreter, as well as an out-of-bounds reach issue in the Content Interpreter. And there was a problem with how the update was tested and deployed.
“Sensors that received the new version of Channel File 291 carrying the problematic content were exposed to a latent out-of-bounds read issue in the Content Interpreter. At the next IPC notification from the operating system, the new IPC Template Instances were evaluated, specifying a comparison against the 21st input value. The Content Interpreter expected only 20 values,” CrowdStrike said. “Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash.”
While CrowdStrike says this exact scenario will not recur, the company is making changes to its process and mitigating steps to “ensure further enhanced resilience,” the company said. CrowdStrike has also engaged two software security vendors to conduct an extensive review of the Falcon sensor code for security and quality assurance, and an independent review of the end-to-end quality process from development to deployment is underway.
“Owning” Its Mistakes
At the Innovators & Investors Summit at the Black Hat USA conference in Las Vegas, moderator Chenxi Wang kicked off her panel with a question for CrowdStrike CTO George Kurtz: “What happened?” Kurtz apologized to the room  — an action that appeared to be well-received by the audience — and noted the company had released the results of the root cause analysis.
The company acknowledged its failures again a few days later, as CrowdStrike president Michael Sentonas was on hand Saturday at the DEF CON hacker convention to accept the 2024 Pwnie Award for Most Epic Fail. The Pwnie Awards recognizes the most outstanding achievements as well as the greatest failures in cybersecurity over the past year. The Most Epic Fail category is for a “spectacularly epic fail — the kind of fail that lets the entire infosec industry down in its wake,” according to the Pwnie Awards’ description.
The Pwnie Awards said back in July that the massive global outage made CrowdStrike an automatic winner. The impact the outage had globally was highlighted by the fact that CrowdStrike was awarded a two-tiered trophy instead of the traditional small pony-shaped trophies awarded to winners in other categories. Sentonas said the trophy will be displayed at the company headquarters in Austin, Texas, to serve as a reminder to staff that “these things can’t happen.”
“Definitely not the award to be proud of receiving,” Sentonas said in his acceptance speech. “I think the team was surprised when I said straight away that I’d be coming to get it. We got this horribly wrong, we’ve said that a number of different times. It’s super important to own it when you do things well, it’s super important to own it when you do things horribly wrong, which we did in this case.”