Misinformation in malware analysis
Para la versión en castellano, click aquí.
In the last few years we have left the information age and entered the misinformation age. Misinformation is as easy to find and as ubiquitous as information. Recently misinformation has even reached my very niche corner of IT security research: malware analysis.
Malware analysis is something I have been doing for almost all my career so I’m clearly passionate about it. That’s why I have decided to debunk Jonathan Scott’s (known on Twitter as @jonathadata1) research paper titled “Uncovering the Citizen Lab an Analytical and Technical Review Disproving Catalangate”. This paper references Citizen Lab’s research titled “CatalanGate: Extensive Mercenary Spyware Operation against Catalans Using Pegasus and Candiru”.
Catalangate and Pegasus
On April 18, 2022, Citizen Lab has published a research which, in collaboration with Catalan civil society groups, has identified at least 65 individuals targeted or infected with mercenary spyware. Jonathan is extremely interested in Pegasus, so naturally he was aware of this report. He then proceeded to investigate the claims made by Citizen Lab and produced a research paper uploaded to ResearchGate which, in his own word, “debunks” the Citizen Lab report.
Just a quick note before we proceed: ResearchGate is a social network for researchers. It allows any researcher to upload any paper. It shouldn’t be viewed as any kind of a stamp of approval for the paper contents.
Brief note on personal attacks
I do not like the personal attacks Jonathan makes on researchers. He is a very abusive person and tries to viciously attack anyone who doesn’t agree with him. I know, as I’ve been on the receiving end of these attacks.
I will not comment on the first parts of his report, which only serve as a way to try and discredit the people doing the Catalangate Pegasus research, not the research itself. These personal attacks have zero technical value, they are instead very petty and childish.
I will focus solely on the technical aspects of the report.
Misinformation setup: vectors of attack
In his report Jonathan tries to ignore parts of the research into Catalangate in order to simplify the issue in a way that makes his false statements more plausible.
According to the Citizen Lab research on Catalangate:
Victims were infected through at least two vectors: zero-click exploits and malicious SMSes.
The zero-click exploits Citizen Lab mentions are two iMessage exploits and one Whatsapp exploit. These exploits are able to infect devices without any user interactions (hence “zero-click”). Jonathan conveniently ignores iMessage exploits and the SMSes instead trying to frame the conversation as if Citizen Lab reported only on the Whatsapp exploit.
This is clearly a tactic meant to setup the conversation by misrepresenting the point Citizen Lab made and then refuting this misrepresentation. The tactic is so well known that Schopenhauer calls out this behaviour in his The Art of Being Right.
Jonathan specifically states a point that is false just so that he argue with himself:
I want to remind the readers of this white paper that the basis of the claims that Catalonians were hacked with Pegasus spyware come from a vulnerability found by WhatsApp
Refuting the point that wasn’t made
Even when Jonathan tries to refute his new re-framed point he fails to do so. He tries to prove his point that all of the domains committed to Amnesty International GitHub in one pull request are related just to the Whatsapp exploit used in Catalangate. Again, there’s nothing to suggest that all of the domains are connected to this very specific exploit.
Jonathan continues his misrepresentation of the Citizen Lab’s research by writing that the domain had to be active in April or May 2019. Again, this creates an argument that wasn’t made by anyone. Jonathan tries to connect the domains only with the Whatsapp attacks that happened in April and May 2019, even though the Citizen Lab report calls out many different attacks with different timelines.
In a very entertaining twist of events, even though Jonathan completely misrepresented Citizen Lab’s research, he couldn’t even disprove this misrepresentation. For example, Jonathan calls out the domain name redirstats.com, which supposedly was expired in April/May 2019 (and hence disproves the argument no one except for Jonathan has made).
However, based on his own raw data in his own GitHub this domain was active in that period, as you can see in the screenshot below.
As you can see Jonatan has mixed up the date on which the record was archived (May 24, 2019) with the dates that the domain was active between (Nov 2018 to Nov 2019). Clearly, the domain was active in April and May 2019. Hence Jonathan even fails to debunk the misrepresentation of the Citizen Lab’s research. It’s a very ironic twist of events.
The “false positives experiment”
The next section of Jonathan’s paper deals with an experiment for which he wanted to gather 50 volunteers. He managed to only get 9. The experiment setup was very odd indeed.
Jonathan has asked the volunteers to fake Pegasus infection indicators of compromise on their devices and then run a tool which detects these indicators of compromise. Unsurprisingly, tool designed to detect an indicator of compromise detected the indicator of compromise. I’m not sure what Jonathan exactly wanted to prove, but I can guess.
Apparently Jonathan’s argument goes like this. If you can fake an infection by sending a message with an indicator of compromise, it proves that indicators of compromise are worthless. I’m struggling to understand this logic. The tool which detects specific domain names detected these domain names. Indicators of compromise are used in a specific context, in order to detect attacks on likely targets. These detections are then analysed in detail to confirm the infection.
His whole experiment is based on the use of the Mobile Verification Toolkit. This toolkit is comes with a very specific warning:
Warning: MVT is a forensic research tool intended for technologists and investigators. Using it requires understanding the basics of forensic analysis and using command-line tools. This is not intended for end-user self-assessment. If you are concerned with the security of your device please seek expert assistance.
Getting the automated results of forensic analysis is a starting point for any forensic investigation. It’s the beginning of the long and complex process, not the end of it.
Jonathan’s logic implicitly assumes that the targets of attacks knew the indicators of compromise during the attack and that they faked the infections. He then, with this assumption, proves that the attacks are faked. He has proved a tautology: if attacks are faked, they are faked. He presents no evidence that the actual attacks, the ones from the Citizen Lab’s research, were in any way faked. He only presents the experiment which confirms that fake attacks he orchestrated are fake.
Additionally, he again provides more entertainment by failing to properly conduct the experiment. Instead of having 50 volunteers (as he set out to do), he only managed to get 9. Out of these 9, only 7 completed his instructions successfully (again, not proving anything of substance). Jonathan has messed up proving a tautology.
The “analysis” of victims
This section of the paper makes a very interesting statement. This statement seems to be that if Jonathan cannot himself confirm infection with spyware then the infection didn’t happen. He proceeds to say that he’s not happy with the fact that some victims lack indicators of compromise or detailed timelines.
I won’t be arguing with this point, because it’s clearly non-scientific and biased.
Is he doing it with a malicious intent?
The question you inevitably will ask yourself now is: does Jonathan have a malicious intent or is he just not very good at malware analysis? I believe it’s the former, based on the way he operates. Whenever someone tries to correct his findings in public his immediate response is to discredit that person. You can see an example of that in the Twitter thread below.
He is so aggressive towards others that Twitter investigation into reported content found that he was threatening to hack or expose another person’s personal information, more than once.
I also believe that he has malicious intent due to my own experiences. Previously, when I reached out to him in DM on Twitter and tried to correct some of his findings in private he pretended that my corrections do not matter.
Since Jonathan didn’t seem to care about sample 1 I have reported sample 1 as false positive. Jonathan has lost it. He tried to humiliate me by pretending that I have a groundbreaking report on Pegasus that I will be releasing and tagging media and journalists that reported on him or interviewed him.
I know from conversations with others that I’m not the only researcher who has this impression. This shows that Jonathan doesn’t care about the truth. The only thing he cares about is whether his misinformation campaign is publicly challenged.
Why not just ignore him?
You may ask “why not simply ignore Jonathan?” Unfortunately, Jonathan is very good at publicising his findings. For example, he was interviewed by the BBC’s Business Daily (they have subsequently issued a correction) and his paper is publicised by many different news websites. His claims were even quoted during the Polish Senate Committee hearing on Pegasus. The tactic of ignoring Jonathan would work only if the media followed suit.
The argument can be made that debunking claims spreads the debunked claims. However, Jonathan and his claims are too influential and dangerous already. Debunking these claims is likely to reach the same, if not even smaller, audience.
I also believe he is dangerously and on purpose spreading misinformation and the media is happy to provide an outlet for him. It’s very dangerous and it, simply put, bothers me. My hope is that this post will help all of us understand how misinformation is created and it will help us spot the signs.
We, all of us, deserve better. We deserve facts, not rhetoric and personal attacks. We deserve information, not misinformation.
