A week after an ill-fated update from cybersecurity giant CrowdStrike knocked out an estimated 8.5 million Windows computers, causing problems ranging from downed medical systems at healthcare facilities to delaying flights for many airlines, there are organizations still trying to restore access to their remaining affected systems.
Healthcare companies are among the most impacted organizations, with the corrupt file affecting about half the members of the Health Information Sharing and Analysis Center, says Errol Weiss, chief security officer for the Health-ISAC. As of July 25, only 18% of affected organizations had fully recovered their systems, while three-quarters of companies still had up to 25% of their systems still needing attention, Weiss says.
Many organizations had Windows-based medical devices, and now they are likely looking at a long-tailed recovery, says Weiss.
“My guess is that a lot of automated remediation was shared on Friday and Saturday—those methods probably helped a lot to get to the majority of completion,” he says, referring to tools and scripts provided by Microsoft, CrowdStrike, and other companies. “But some of those scripts and automated fixes probably won’t work on the kinds of devices that we’re talking about, and now healthcare organizations have to take a manual look.”
Microsoft released a USB Recovery Tool which allows administrators to the option to use a USB drive to recover impacted systems from WinPE or from from safe mode. The tool may recover from safe mode even if BitLocker is enabled on a device and a recovery key is unavailable. There are also detailed recovery steps for Windows clients, servers, and OSes hosted on Hyper-V, as well as affected Windows 365 Cloud PCs and Azure virtual machines.
Measuring Impact of the Outage
On July 25, CrowdStrike estimated that 97% of affected computers have returned to active status, as measured by the state of its Falcon software at the center of the outage. Managed security services provider Quest Software, whose customers span a gamut of sizes, is still offering aid to customers who are working through the issue. The remaining companies likely represent a few hard-to-patch systems at larger firms and a large number of smaller firms, who do not have the technical expertise to easily recover, says Kent Feid, senior director of product management at Quest Software.
“That 3% really represents the number of devices, and so that would equate to likely a substantial amount of small businesses still being impacted who are still somewhat unsure how to attack this,” he says. “Smaller businesses tend to leverage more IT generalist or even don’t staff IT experts in-house.”
The vast impact of the outage has still not been tallied, but insurance services firm Parametrix Solutions estimates that the event impacted a quarter of the Fortune 500 companies, with losses reaching $5.4 billion, including nearly $2 billion in losses for healthcare and more than $1.1 billion for the banking sector.
Even With Tools, Many Companies Worked the Weekend
While the recovery process is, for the most part, fairly simple, technical experts have gauged that each system requires an average of 15 minutes to recover, because it requires an administrator to physically access each system. In addition, companies that used BitLocker to encrypt the hard drive—a cybersecurity best practice, especially on laptop systems—would have to find the encryption key and input that key at the beginning of the process.
“There’s no way to do this remotely because it has to be done in safe mode, where networking isn’t working, so you can’t connect to the machine remotely,” says Vadim Vladimirskiy, CEO of Nerdio, a virtual desktop management firm.
At least 700 outages coincided with the bad update from CrowdStrike, with 39% of outages rated Critical. Source: Parametrix Solutions
Nerdio, whose business is providing virtual desktops for its customers, said its customers were only minimally impacted by the failed update and its cloud desktop systems were easily repaired by recovering to a previous image. While many customers connect to Nerdio’s service using a Windows computer, only systems left on during the 78-minute window—during which the bad CrowdStrike update was distributed—were impacted. Affected customers could just switch to a different system to access their virtual desktop, limiting any impact, Vladimirskiy says.
Ironically, healthcare firms recovered by falling back on measures implemented to protect them from a threat CrowdStrike is deployed to prevent—ransomware. Health-ISAC’s Weiss compiled a list of systems affected by the attack, and it included patient services, lab collections, secure file transfers, dictation and transcription services, shipments, electronic medical records, and Medicaid and insurance billing.
“I started hearing about the impacts to these organizations, and looking at the list, and it’s like, my gosh, this sounds just like another ransomware incident,” he says. “So that’s what was going on inside healthcare on Friday for those organizations that were impacted by this, they just went, ‘Okay, systems are down, we’re going to the manual backup procedures, we’re going to paper,’ and they knew what to do, because they were drilling [their response to ransomware] in the past.”
Preventing the Next Big Failure
The bad update also came after a significant outage of Azure services affected an above average number of companies, according to Parametrix Solutions. (On average, about 300 service disruptions happens among the Fortune 500 every day, the firm said. On Thursday, July 18, 419 coincided with the Azure outage, and on Friday, at least 700 occurred as company dealt with the bad update from CrowdStrike.)
While CrowdStrike is feeling the market’s wrath right now, the company will likely not be down long, because businesses need the type of services the company—and others like it—provide, says Quest Software’s Feid.
“No software development company—I include ourselves in that—is perfect, right?” he says. “What’s hard, I think, especially in the security industry, and specifically for a company like CrowdStrike, they are being looked at and relied upon across a large part of the market to protect endpoints. … and the product is specifically designed to be ahead of the curve as much as it can, which means you can’t have it both ways, consumers—there’s always gonna be inherent risk.”