On July 19, 2024, CrowdStrike rolled out a “content update” to its customers running the CrowdStrike Falcon endpoint agent on Windows devices, resulting in disruption to organizations worldwide in multiple industries, including travel, banking, healthcare, and retail.
Threat actors commonly use large scale disruptions and incidents as opportunities to take advantage of victims. In this post, we provide clarity on Sophos’ understanding of what happened, and answer key follow-up questions from our customers and partners.
The goal of all companies in the cybersecurity space, Sophos and competitors alike, is to keep organizations safe and protect them from attackers. While we compete with one another on the commercial stage, we are – most importantly – a community united against cybercriminals as a common enemy. We extend our peer support to CrowdStrike at this time and wish every affected organization a swift recovery and return to normalcy.
Cybersecurity is an incredibly complex, rapidly evolving landscape. “For those of us with the skin-in-the-game of living in the kernel, it’s probably happened to us at one time or another, and whatever precautionary steps we take, we are never 100% immune” said Joe Levy, CEO of Sophos, on LinkedIn.
Issue summary
- This was not the result of a security incident at CrowdStrike and was not a cyberattack.
- Although it was not the result of a security incident, cybersecurity consists of confidentiality, integrity, and availability. Availability was clearly impacted, so this is categorically a cybersecurity failure.
- The issue, which resulted in a blue-screen-of-death (BSOD) on Windows machines, was caused by a product “content” update rolled out to CrowdStrike customers.
- Organizations running CrowdStrike Falcon agents on Windows computers and servers may have been impacted. Linux and macOS devices were not affected by this incident.
- CrowdStrike identified the content deployment related to this issue and reverted those changes. Remediation guidance has been issued to CrowdStrike customers.
A note about “content” updates
This was a typical product “content” update to CrowdStrike’s endpoint security software—the type of update that many software providers (including Sophos) need to make regularly.
Content updates, sometimes called protection updates, improve an endpoint security product’s protection logic and its ability to detect the latest threats. On this occasion, a content update from CrowdStrike had significant unforeseen consequences. However, no software provider is infallible, and issues such as this can (and do) affect other vendors, regardless of industry.
CrowdStrike response
CrowdStrike has issued a statement on its website with remediation guidance for its customers. If you are affected by the issue or receive inquiries from your customers who use CrowdStrike, please refer to this official CrowdStrike page:
https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
As always, vigilance is critical. Cybercriminals are registering potentially malicious domains (typo-squatting) and using “CrowdStrike remediation” in phishing campaigns to try to take advantage of victims. If you contact or are contacted by CrowdStrike, please validate that you are talking with an authorized representative.
Were Sophos customers impacted by the CrowdStrike incident?
Customers using Sophos for endpoint protection, including those using Sophos Endpoint with Sophos XDR or Sophos MDR, were unaffected. A small number of customers who use the Sophos “XDR Sensor” agent (available with Sophos XDR and Sophos MDR) as an overlay on top of CrowdStrike Falcon may have been affected.
What does Sophos do to mitigate the risk of having a similar service disruption?
Every endpoint protection product, including Sophos Endpoint, provides regular product updates and continually publishes protection (content) updates. Threats adapt rapidly, and timely protection logic updates are essential to keep up with the constantly evolving threat landscape.
Having provided leading endpoint protection solutions for over three decades, and learning many lessons from past Sophos and industry incidents, Sophos has robust processes and procedures to mitigate the risk of customer disruption. However, that risk is never zero.
At Sophos, all product updates are tested in internal, purpose-built quality assurance environments before being released into production. Once in production, product updates are released internally to all Sophos employees and infrastructure worldwide.
Only once all internal testing is complete, and we are satisfied that the update meets the quality criteria, will the update be gradually released to customers. The release will start slowly, increasing in velocity, and staggered across the customer base. Telemetry is collected and analyzed in real time. If there is an issue with an update, only a small number of systems will be affected, and Sophos can roll back very quickly.
Customers can optionally control Sophos Endpoint product updates (not protection updates) using update management policy settings. Software package options include Recommended (Sophos-managed), Fixed-term support, and Long-term support, with the ability to schedule the day and time when updates should occur.
As with product updates, all Sophos Endpoint content updates are tested in our quality assurance environments before they are released into production, with each release reviewed to ensure that it meets our quality standards. Content releases to customers are staged as part of our ongoing QA controls and we monitor and adjust releases based on telemetry as necessary.
Sophos follows a secure development lifecycle to ensure our solutions are built securely and efficiently, detailed in the Sophos Trust Center. Additional information on the release and development principles for Sophos Endpoint can be found in our knowledgebase.