Triangulation: Did “the NSA” fail to learn the lessons of NSO?
The 1st of June 2023 saw perhaps the most exciting development in the targeted spyware research space in recent memory.
The Russian government’s Federal Security Service (FSB) released a statement indicating that they had detected a hacking campaign by the US National Security Agency (NSA) that compromised “thousands” of Apple iPhones within Russia. The FSB’s statement also, bizarrely, asserted collusion between the NSA and Apple, throwing the veracity of the rest of the FSB’s claims into question. Private sector offensive actors like NSO, QuaDream, and others, have regularly fielded zero-click, zero-day exploits against Apple and Android phones, without any collusion with these companies.
The FSB statement alone might have barely registered a blip in the news cycle. However, around the same time, Kaspersky Lab released a public report about forensic traces left behind by the very same hacking campaign, which they called TRIANGULATION. The technical details that Kaspersky published underline the fact that a bona fide hacking campaign existed and was detected, even if the jury is still out on some of the particulars. Interestingly, Kaspersky’s report leaves unsaid how the campaign was initially discovered (and who discovered it), but analysis of the domain names published by Kaspersky indicates that a defender may have discovered the attacks in early-to-mid 2021, causing the attacker to rotate some of their infrastructure.
For their part, Kaspersky told Reuters they came to know about the attack in early 2023, when they discovered suspicious data exfiltration from devices used by Kaspersky’s management. Once Kaspersky had identified dates and times of interest, they examined forensic timelines of these same mobile devices, and noticed a number of items they regarded as suspicious, including a process referring to itself as “BackupAgent” running on the phone around the times of compromise.
While this story is presently confined to Russia, it seems poised to break through onto the global stage. If, indeed, the attacker is a US government agency as the FSB claims, and if, indeed, this attacker has targeted thousands of devices in Russia, it then stands to reason that the scope of the activity extends well beyond Russia. The tantalizing prospect here is that there are a half-decade’s worth of targets to discover around the globe (judging by registration and expiry dates of published domains, 2019-present), with the attacker’s target selection unfettered by the sort of geographic restrictions that might be imposed by a commercial surveillance vendor. (As is well-known, NSO Group reportedly bans many Pegasus users from spying in Israel, Iran, Russia, China and the United States).
Unanswered Questions: What is “BackupAgent” telling us?
Kaspersky has released a utility for command-line whizzes called “Triangle Check” that, more or less, flags traces of a process referring to itself as “BackupAgent” consuming mobile data. The utility also displays a result of “SUSPICION” if a heuristic developed by Kaspersky is matched, though the interpretation of the “SUSPICION” result is unclear. Perhaps, as Kaspersky obtains additional data from compromised devices, they will hone their tool and add additional indicators.
At first glance, the “BackupAgent” indicator seems rather compelling. While a legitimate binary called BackupAgent is still shipped with the latest iOS version as of this writing (16.5 — you can find it at /usr/libexec/BackupAgent on your iPhone), it is unclear what might cause this process to be invoked during normal phone usage, given that another process, BackupAgent2, has apparently subsumed its functionality. The BackupAgent binary may manage some deprecated or legacy aspect of conducting an “iTunes backup” or “restore” of the device while it is connected to a host computer. Even if the process happens to be invoked, the prospect of the legitimate BackupAgent consuming mobile data seems hard to fathom.
There are at least four interesting questions about any proposed indicator of compromise, such as “BackupAgent”.
1. Is this indicator definitive evidence of compromise?
If indeed it is not plausible for a legitimate process calling itself “BackupAgent” to consume mobile data, and the backup in question has not been tampered with, then, yes, it seems this is a definitive result indicating some sort of compromise. If, on the other hand, there is actually some weird corner case where a legitimate process calling itself “BackupAgent” can consume mobile data, then no, presence of this indicator is not definitive.
It could be instructive here to perform an analysis of various ways that the legitimate BackupAgent could be triggered and use data.
2. If definitive, is this indicator specific?
In other words, if I see that “BackupAgent” has consumed mobile data on my phone, can I conclude that I was targeted by TRIANGULATION? Or might there be a totally unrelated group (for fun let’s call it QUADRILATERAL) that, through a stroke of ad-hoc genius (or shared lineage of tooling), also happened upon the “BackupAgent” process name?
A situation where two threat actors accidentally collide on the same legitimate iOS process name seems conceptually far more likely than a situation where two actors plumb the depths of their creativity to generate a “fake” process name and achieve an identical result. (Note that NSO Group famously made use of “fake” process names, and these “fake” names turned out to be an excellent fingerprint for detecting traces of older Pegasus infections).
Indeed, the possibility of overlap on legitimate process names is why we at Citizen Lab concluded in a recent report only with “medium confidence” that the presence of the QuaDream process name /private/var/db/com.apple.xpc.roleaccountd.staging/subridged when observed alone was indicative of infection with QuaDream spyware (“subridged” is also a legitimate Apple process name, though the legitimate process would never be run from this directory). Only in cases where additional indicators linkable to QuaDream’s spyware were present did we upgrade our assessment to “high confidence.”
3. If “BackupAgent” is specific, what is it specific to?
Is it a type of mercenary spyware sold to multiple governments, like Pegasus? Is it a type of spyware operationalized by multiple US Government entities (ala the “Equation Group Implants”)? Is it spyware developed from the ground up by a single unit of a single agency for its own exclusive use? Or is it something else entirely?
4. What sort of completeness should we expect from this indicator?
While security professionals do a good job making clear that the absence of an indicator is a “nothing found” result rather than a “clean bill of health” result, there is still an interesting question about what mental model we should bring to the table about the “BackupAgent” indicator. Will this cover 100% of TRIANGULATION cases? 50%, 10%?
Could TRIANGULATION have been “smarter”?
At the end of the day, the TRIANGULATION attack was detected. Could attackers have avoided this outcome through more clever operational design? Is avoiding detection even important?
In this regard, it may be instructive to study the case of NSO Group, a threat actor that continues to field zero-click, zero-day exploits targeting the latest devices, despite a raft of negative publicity stemming from their customers’ abuse of Pegasus to target journalists, human rights defenders, activists, political opposition, and others. NSO Group has had the benefit of rapid feedback in the form of a series of reports from us at Citizen Lab, as well as Amnesty International and other research groups, from 2016 through 2023. These reports have described not only abuse of Pegasus but also how we detected and attributed Pegasus.
Here are some of the lessons NSO Group have likely learned over the years, both from these reports, and from other cases where their spyware was detected.
Lesson #1: You don’t have to be at the top of your game to detect sophisticated mobile spyware.
With apologies to Turkey, that government is not generally regarded as being among the ranks of “top tier” cyber powers. Yet, they still managed to detect a Saudi-directed Pegasus hack against an (as yet unnamed) target sometime in late 2019. How do we know? Well, the Turkish CERT (quietly) added domains used by the Saudi Pegasus customer in 2019 to its “list of malicious links”, which appears intended for use among cyber defenders in Turkey. The Turkish CERT also sinkholed these domains nationwide, pointing them to a Turkish CERT IP address to gather victim telemetry. However, as this sinkholing was cleverly conducted only on network infrastructure within Turkey (via network injection and local manipulation of DNS answers) it would have been non-obvious to those outside Turkey that this action had been taken.
You also don’t need to be a signals intelligence whiz to detect this sort of exfiltration. As we at Citizen Lab published, we detected several zero-click Pegasus infections in 2020 by flagging large uploads to unpopular domain names from mobile phones. It turns out that this sort of naïve analysis is quite tractable and productive for mobile devices, especially if the users of the devices are playing within the confines of Apple or Google’s walled garden.
Lesson #2: You don’t have to be a whiz to uncover a hidden infrastructure.
No matter how much an attacker thinks they are varying the behavior of their infrastructure, it is likely that there will be some link between operations carried out with shared tooling in multiple contexts. Just as attackers perpetually maintain a cyber advantage over defenders because attackers only have to find one weakness whereas defenders have to secure all avenues, fingerprinters (those defenders seeking to track attacker infrastructure) perpetually maintain a cyber advantage over attackers because fingerprinters only have to find one commonality among infrastructure, while attackers have to ensure no commonality is findable.
The Iranian government famously arrested CIA assets active in their country after unmasking a hidden communications network, in which each asset communicated with Agency handlers via a separate CIA-designed website. Despite Agency efforts to differentiate the websites, Iran was still able to discover commonalities shared among the websites, allowing the government to reportedly unravel the CIA’s network within Iran. Iran is also said to have discovered similar CIA-designed websites used by assets in other countries and tipped off those governments to the espionage.
In 2022, we at Citizen Lab replicated what we suspect may have been the Iranians’ analysis in order to understand and illustrate the Achilles’ heel of the CIA’s infrastructure. Based on one website provided to us by Reuters reporter Joel Schectman, we identified 885 websites in at least 29 languages, geared towards suspected Agency assets in at least 36 countries.
The cybersecurity community knows a lot of ways that infrastructure can be linked together. Search engines for Internet scanning data, such as Censys, RiskIQ, or Shodan, expose many popular pivots, including HTTP response hashes, JARM fingerprints, TLS certificates, HTTP header ordering, and so on. But these pivots merely scratch the surface of what is possible for a committed fingerprinter to achieve. What the cybersecurity community does not know is how to design unlinkable infrastructure.
Attacker Takeaway: Is the only winning move not to play?
Let’s say you’re an attacker. Given Lessons #1 and #2, perhaps the only winning move for you is (to the extent that you can) refusing to exfiltrate data from your targets, and refusing to set up infrastructure.
What if you don’t exfiltrate?
Well, if you’re an attacker, you have to exfiltrate something, right? But what if the thing you exfiltrate could be really small, like passwords, login cookies, two-factor codes, or tokens to access the victim’s online services?
Commercial vendors have figured this out, with QuaDream offering a solution to steal passwords and two-factor login codes from devices, and NSO reportedly offering something similar. In other words, wherever possible, you should steal the copy of the information you want “from the cloud” rather than “from the phone”. While breaking into cloud accounts is subject to potential discovery by cloud service providers, it has the potential to leave a device largely exfiltration-free, robbing defenders of a crucial opportunity to flag malicious behavior.
What if you don’t set up infrastructure?
Let’s take it a step further. What if you as an attacker could exfiltrate this minimal payload of passwords and login tokens through someone else’s infrastructure? Say, via an iMessage? Spyware without publicly addressable infrastructure: perhaps the holy grail!
However, for the sake of argument, there might also be a need for you to exfiltrate large items from the device itself, and you might be nervous about conducting these sort of transfers through messaging services where such usage patterns might cause your account(s) to be flagged for abuse or scrutiny, perhaps leading to eventual discovery.
Still, there is no reason that communication with a particular IP or domain name need indicate that a device is compromised. Why not use a popular cloud storage service? At a minimum, the IPs your victims talk to will be shared among a multitude of benign uses. And if you’re lucky, and that cloud storage service allows user uploads to their root domain, the domain name won’t give anything away either: the bucket identifier will be safely ensconced in a TLS encrypted (and with certificate pinning, authenticated) session. Also, using a unique subdomain of someone else’s domain name could become less of a worry for attackers going forward, especially with the advent of TLS 1.3, encrypted DNS, and ESNI, which could place precise information about endpoints accessed by a compromised device outside the reach of some defenders.
Alternative takeaway: What if you don’t care?
Of course, there is an alternative takeaway for sophisticated operators: stealth isn’t everything. Despite NSO learning the lessons of NSO, their attacks continue to be (eventually) detected, and Pegasus victims regularly receive warnings from Apple of possible state-sponsored compromise. Nevertheless, NSO’s latest two zero-clicks, FINDMYPWN and PWNYOURHOME, both apparently leveraging the same (or similar) underlying vulnerabilities in iMessage, were deployed as zero-days for months against several versions of iOS between 15.5 and 16.0.3.
After all, is it that bad if your adversary figures out that “somebody” has exfiltrated terabytes of data from thousands of their devices, after the fact? At the end of Ocean’s Eleven, the casino owners discovered they’d been had (apologies for the spoiler), but this realization did not negate the thieves’ success.
At the end of our “NSA” story, the attackers have the data, and the defenders probably have no (forensic) ability to figure out what the attackers took. Furthermore, the defenders might not even know who, specifically, the attackers are, even if they may have the general sense that the attackers are some US Government agency. Undeterred, the attackers could come back when they discover a new vulnerability or vector.
Obviously, we do not know what is to play out in private, or what is yet to come. Things could get especially interesting if the community of forensic analysts can establish that some US Government agency abused spyware to target journalists or civil society, or if US-allied governments can establish targeting of their personnel, (ala the famous Merkel case). But, there are not (yet) any public indications that the FSB (or anyone) has more than a guess at attribution of this group. It is unclear yet if any spyware or exploits were harmed (i.e., captured or burned) in the commission of this hacking. Is that not an unmitigated success? Perhaps. Time will tell. Until then, we count down to the next zero-click…
Takeaways for Defenders
But let’s say you’re not an attacker. After all, there are probably an order (or several) of magnitude more defenders than the type of attackers we’re talking about here. While we are still in the early days of reports about TRIANGULATION, there are still a number of useful takeaways here for defenders.
Messaging apps are a royal road onto devices.
Like many of NSO and QuaDream’s zero-click attacks we at the Citizen Lab have investigated, this attack appears to leverage a flaw in a messaging app. Past NSO attacks have exploited flaws in iMessage attachment processing (as TRIANGULATION apparently did) as well a flaws in WhatsApp’s negotiation of end-to-end encryption for video calls.
Messaging apps are designed to make it easy for users to find each other, and reduce friction around first contacts, making calls, and sharing information. They are also constantly running on most phones. These factors make messaging apps easy to use, but also especially attractive for attackers.
Could disabling the most-recently-exploited app be enough to protect you? As Eugene Kaspersky tweeted, disabling iMessage would have blocked the attack. And it’s true, if you disabled all your messaging apps (or perhaps eschewed smartphone use entirely) you’d be much less at risk. But you’d also be much less effective in your work and probably less connected with your friends and loved ones.
For most people, disabling the messaging app their friends & colleagues use is not an option.
Lockdown Mode?
Without further knowledge of the specific TRIANGULATION vulnerabilities, it is not fully clear whether or not Lockdown Mode (LDM) would have prevented these attacks (had they been targeted at devices running iOS 16, in which LDM was introduced).
However, after an extended period of observing high-risk users with LDM turned on, we know it is a great feature for users that face higher risks because of who they are or what they do. It substantially increases costs on many categories of attacker. We were even able to confirm that it blocked NSO Group’s iOS 16 zero-click, PWNYOURHOME.
While Lockdown Mode is expected to reduce the zero-click attack surface through iMessage (by avoiding automatic processing of potentially risky messages and attachments from senders unknown to you), it is less clear to what extent (if at all) it would reduce the zero-click attack surface through third-party messaging apps.
Watch the traffic.
As long as sophisticated threat actors are targeting and exfiltrating from mobile devices, defenders need to carefully examine Internet traffic for high-risk users. Even a naïve analysis of mobile network traffic data can illuminate sophisticated threats.
Anticipate change.
The “P” in APT stands for “persistent”. Attackers will attack again. So too must you as a defender be persistent. With every report about a novel mobile attack, attackers have an opportunity to adapt their tactics and try again. If you carefully review attackers’ past behavior and strive to discover what lessons they take away from being detected, perhaps you will be a more effective defender when then next zero-click is unleashed upon the world.
Acknowledgements
Thanks to John Scott-Railton, Emma Lyon, and Ronald Deibert for helpful feedback. Thanks to Angela Chih for graphic design assistance.
