Sophos is not the first cybersecurity vendor to find its perimeter products the target of sustained nation-state attack. If anything is special about the series of events we reveal in “Pacific Rim: Inside the Counter-Offensive—The TTPs Used to Neutralize China-Based Threats“, it’s that we are reporting this hunt / counter-hunt activity as fully as ongoing investigations allow to illustrate precisely what the security industry is facing in terms of the determination and aggressiveness of certain attackers. Through it, we’ve learned a great deal about countermeasures. This essay presents three sets of observations that other defenders can apply.
To raise the adversary’s cost, burn the adversary’s capability. Sophos is large enough to be able to muster serious resources in emergencies, but still nimble enough to respond rapidly and creatively to put the hurt on an attacker. In this situation, we had the home-field advantage of firewalls being relatively predictable environments. Compared to activity on general-purpose endpoints, attackers are compelled to work harder to be quiet and unobtrusive on firewalls. Measure that against the general high target value of firewalls – powerful Linux devices, always on, good connectivity, situated by their nature in trusted places on the network – and you can see both why an attacker would wish to be there and why we were able to meet the attacker effectively on that field.
To be sure, there were a few extraordinary (and tense) moments as we watched the attacker evolving their creative abilities; the UEFI bootkit – we believe it to be the very first observed instance of a bootkit utilized for persistence on firewalls – comes to mind. But that sort of creativity comes at a high cost. A world in which attackers are compelled to find ways to dwell in memory and use UEFI bootkits for persistence is a world in which most defenders would, again, say they had a home-field advantage. (And then they can get on with the process of detecting and responding to those very specific tactics.)
Telemetry has been a major factor in our home-field advantage since the start of activity. One of our first actions early in the Asnarök activity (spring 2020) was to issue an automatically deployed hotfix to not only patch the CVE-2020-12271 bug but to improve fleet-wide observability, increasing the volume and the types of telemetry returned to us for analysis. In the years that followed, telemetry, and the associated detection-and-response processes, became an important pillar of our Product Security program. Privacy concerns were of course front-and-center in our thinking (even though the sort of technical internal system data we needed didn’t touch, for instance, PII), so balancing those concerns and the customer-safety benefits of increased data collection was a painstaking process, especially as law enforcement became involved.
Of course, defending devices that are on-premises in customer environments has its own constraints. In many cases, those take the form of outdated firmware or end-of-life hardware that’s still in “use” far beyond actual usefulness. The second lesson learned in the course of this series of investigations may seem anti-end-user or unenforceable, but in 2024, it bears serious discussion.
For the good of users and of the internet at large, both hotfixes and end-of-life must become non-optional for firewalls. A firewall that’s purchased and then not updated for five years is, frankly, no longer a firewall. A firewall so old it cannot take new hotfixes is, frankly, no longer a firewall.
There’s a lively discussion to be had around end-of-life issues with hardware, but let’s take up the hotfix question first. We know that many administrators, particularly those who still adhere to habits and practices developed in the boxed-software era, are wary of applying patches that they have not themselves tested (even though the *-as-a-Service era has smoothed that process to a large degree). Though we agree that hands-on attention to patches and hotfixes is fair and justified for many other devices on production systems, we argue that firewalls administrators need to recognize the time-criticality of updates to these highly specialized systems, and to trust their vendor to rapidly fix issues for them. Of course, this trust must be earned; recent events have made crystal-clear the seriousness of trusting automatic updates, especially for highly privileged applications. Vendors need to take their updating responsibility seriously with rigorous testing, staggered deployments, transparency into all changes and, critically, detection and response processes built to ensure they can react in a way that materially reduces harm across their customer base.
Over time, as the internet evolves, even the most diligently updated hardware will reach the end of its ability to cost-effectively support necessary updates and features. At some point, these older devices become not just dead, but actively undead and dangerous, as the events described in Pacific Rim timeline: Information for defenders from a braid of interlocking attack campaigns show all too well. The firewall becomes a kind of “digital detritus,” the hardware equivalent of the old and unattended data described by Jillian Burrowes many years ago – out of date and destined to be abused. A conversation about how to reduce the attack surface such devices present is a difficult, large, and important conversation – one we believe our vendor community, and the larger defender community, should undertake sooner rather than later.
Security is a team sport. Offense is a team effort. Defense needs to be a team effort. And “team” is the operative, the necessary scope here. Sophos’ story is everyone’s story. Not only are we not the only targets, evidence (both public and more closely held) indicates that we’re not even the infosec concern getting the worst of it. As our story shows, the attacks on our perimeter devices were a multi-faceted team effort, the methods of ingress and persistence passed around from criminal group to criminal group. To even the sides, businesses must seek communion with industry peers, with government and law-enforcement entities, and even with independent or even anonymous security researchers. Companies based in Europe and the West may find the structures for public-private relationships far different from those in nations such as China, but this is a rally cry for all of us to leverage our collective intelligence to fight back.
In the course of these events, we have worked with a great number and variety of government partners; we list a number of them at the end of the main article. Sophos participates in organizations such as JCDC because it’s the right thing to do, but in the last couple of years we are increasingly seeing real benefits, real information sharing, real analysis, real muscle put into takedowns. As momentum builds, defenders need to find the most effective ways their organizations can take a seat at the table(s) that make sense for their businesses. As our saga shows, the adversaries don’t hesitate.
But the fellowship of defenders isn’t just for those with badges or business cards. Bug bounties – once controversial, and still under-appreciated as a form of defender cooperation – also play a part in a strong defender community. On multiple occasions in the course of these events, we paid bounties to researchers reporting vulnerabilities similar, or identical to, those found to be in use by the attacker(s). In at least one case the reported vulnerability was already being used against high-value targets, leading to potential questions of how that happened and how the researcher might have been related to the attackers.
Here’s our answer to those questions: Who cares. Do we know how, or if, the researcher and the attacker(s) are related? No. Can we? Highly unlikely. Does it matter? Not really – the only thing that’s important, and the thing that makes it worth it to have paid the bounty, is that we were able to significantly disrupt an ongoing operation and help victims recover from a serious attack. How many more victims could the adversary have compromised, had the issue (CVE-2022-1040) not come to our attention via our bug bounty program?
As detailed elsewhere, this saga continues. The wheels of law enforcement sometimes grind slowly, and the entities we believe to be behind this multi-year effort are still very much active. (Indeed, global conflicts have become far more complicated since this all started half a decade ago.) Inside Sophos, the multi-team efforts required to quickly parry waves of attacks have led us to refine and improve in-house processes here – some large, some very small. Those improvements are also an ongoing process.
We now make our case to the rest of the industry: Join us in working to raise adversaries’ costs by burning their capability; to find a way to sweep away security detritus that once helped to protect the internet, but now only hurts it; and in treating cyber-defense as a team effort, as the adversaries do.
Sophos X-Ops is happy to collaborate with others and share additional detailed IOCs on a case-by-case basis. Contact us via pacific_rim[@]sophos.com.
For the full story, please see our landing page: Sophos Pacific Rim: Sophos defensive and counter-offensive operation with nation-state adversaries in China.