Factsheet: profanity vulnerability report
QANplatform publishes the following document for the sake of transparency and educational purposes about the recently occurred event of the profanity vulnerability exploitation.
It is easy enough to inspect and evaluate the chain of events after they happened. This document’s sole purpose is to demonstrate whether there was any chance to avoid the incident (and if yes with what probability) given that prior decision-making events were made based only on information available at that time when those particular decisions were made.
Our goal is that by transparently disclosing and evaluating the said chain of events we can transfer additional knowledge to the crypto community so that we can collectively learn from it.
Update: February 22, 2023
We are pleased to announce the commencement of the QANX Token relaunch process.
1.) 27 October 2021: The bridge smart contract gets audited
The bridge smart contract in question was audited by CertiK. The audit contains a single partially resolved point (QAN-01), which is identified as a “centralization risk” with a rating of “significant”.
It is important to point out that ALL bridges in the crypto world are more centralized than the chains they themselves connect, this is by design and is the case with every other bridge out there, it is not unique in any way. This fact is confirmed by the audit itself. The QANX bridge is fully centralized in the sense that there is one wallet that is authorized to move the entire token volume allocated to the bridge.
This functionality was needed for the following reasons:
- The migration to the mainnet will probably require new bridge technology
- A more decentralized bridge alone would require several years of development and/or would incur irrationally high costs, which was not an objective and was not affordable at that stage of the project lifecycle
- Multi-party signatures (so-called multisig) generally increase transaction fees for bridge users
- However, these multisig solutions do not provide any additional protection given proper key management (and the problem was not related to key management in any way)
2.) 28 October 2021: Vanity contract address generation
On this day, the private key (which was later compromised as detailed in step 8.)) for deploying the bridge smart contract was generated. The steps of this process are outlined below:
2.1.) Google search for “GPU Ethereum vanity” keywords
The assigned developer searched for this term because this was literally the job to accomplish, namely to generate private keys capable of deploying easily distinguishable smart contract addresses on the Ethereum platform using GPU acceleration. Even today, this search yields the software used by the QAN development team as a first hit on the result list. As the software found was perfectly suited to the desired goal, the developer did not search for further alternatives.
Obviously it is impossible to search for a name of the software whose name is not yet known to the person performing the search, since that is the exact piece of information we are looking for in the first place. One can search for a desired goal, which search attempt will hopefully yield the name of the software capable of performing that desired goal as a result.
For this particular purpose the search engine does not list the more popular software called “johguse/profanity” as a primary result, but a derivative (so-called fork) named “cenut/vanity-eth-gpu”. Although it is still uncertain up until today what the root cause of this behavior is, but most likely the fact that the name of the original software does not reflect the purpose of the software in any way, while the name of the derivative does (profanity vs. vanity).
2.2.) Downloading the software and inspecting it in-depth
The developer downloaded the open source wallet generator software and inspected it for any presence of malicious code (“calling home”, pregenerated key list etc.). No malicious code elements were found by the developer, since the software did not contain any bits of malicious code indeed. The developer was not at fault.
Following the inspection conducted by the developer an information security specialist also inspected the sources and his findings concluded the same report, no malicious code was found.
It is important to highlight that it became apparent that the vulnerability which was disclosed later (and which is discussed in this factsheet in point 7.1.)) was not detected by the development and security specialists of other crypto projects handling $5 billion+ value (e.g. Curve, Wintermute, IndexedFinance, The SolaVerse, ParaSwap etc.) in time either unfortunately, it was such an insidious flaw.
2.3.) Applying a zero-trust policy irrespective of the outcome of the investigation
Although the internal investigation did not reveal any malicious code fragment in the key generation software (since it was not present, the source of the error was different), the development team still performed the key generation on a computer physically isolated from any network (so-called “airgapped” machine), thus physically eliminating the possibility of a key leak.
2.4.) Persisting the generated key
Once a private key capable of generating a contract address in the desired format has been generated, it is stored exclusively on a FIPS 140–2 Level 3 compliant storage medium (no more physical storage can ever be placed in this computer, this is also physically impossible).
Subsequently, the wallet address associated with the private key was also saved in a central wallet registry with the name of the software used to generate the key.
3.) 29 October 2021: Uploading and configuring the smart contract on testnets
The details of this step are the same as in step 5.), the only difference is the target network (testnet/mainnet).
4.) 30 October 2021: Generating a signature key with applied limits
A secondary key was also created, but for a completely different purpose. Important properties of this particular secondary key:
- It is only allowed to sign transactions up to a limited amount (5.000.000 tokens)
- It is not allowed to grant further signature rights to other addresses
- It must be available on an Internet-connected machine, as it must sign bridge transactions in real-time
The key was generated on the same airgapped machine disconnected from the network, but with a different software (geth), since this key did not have to produce a vanity address.
5.) 30 October 2021: Deploying the smart contracts onto the mainnets
5.1.) Signing and broadcasting deployment transactions
The smart contract deployment transactions to the Ethereum and BSC mainnets were crafted and signed completely offline on the same airgapped machine which was previously used for key generation.
The signed serialized transactions were transferred via another storage medium with the same FIPS classification to an internet-connected machine.
Above signed serialized transactions were broadcasted to the respective blockchain networks and the smart contract was successfully deployed on both networks with the desired vanity smart contract address.
As a result the private key generated in step 2.3.) became the owner of the deployed smart contract. Since this role maintains high privileges, the key itself was stored in a physically isolated safe deposit box on a FIPS 140–2 Level 3 classified storage medium following advice suggested in the audit result.
In this respect, we have also followed the CertiK audit recommendations to maximize the security of key management and make it mathematically impossible for keys to leak.
5.2.) Configuring the deployed smart contract
After successful deployment the smart contract was configured. The configuration-related transactions were signed, transferred and broadcasted along the same logical steps as the deployment transactions themselves.
The configuration transactions were successfully processed on-chain and resulted in the following modifications:
- A wallet address permitted to sign up to 5.000.000 tokens was set
- The transaction fee percentage was set
From this moment on the bridge was operational.
6.) 8 May 2022: Bridge hack: parameter validation error
On this day, a parameter validation flaw in the signing module allowed an attacker to withdraw tokens on one side of the bridge without depositing the same amount of tokens on the other side of the bridge.
Once again, it is important to note that this particular event has NOTHING to do with the events of 11 October 2022 detailed in point 8.).
Summary of events following the exploitation of the parameter validation error:
6.1.) Shutdown of the bridge’s components
The public (web interface) and private (signing module) components have been shut down.
It is important to highlight that the private key of the signing module (which was generated in step 4.) and was granted signing rights up to 5.000.000 tokens as described in step 5.2.)) was still allowed to sign transactions, even though the signing module was disabled.
However, this did not cause any problem, as the key could not be physically compromised as it was hosted in a hardware wallet developed in-house.
Nevertheless, it was decided that the signing rights should be taken away, so it was necessary to:
- to access the key generated in step 2.)
- on the computer used in point 2.3.) in a similar way to point 5.1.):
- sign the transactions blocking access of the limited signer
- physically transfer them
- broadcast them to the respective networks (ETH+BSC)
During these steps the development team also followed the proposed secure protocol to the fullest extent, there was physically no chance of key leakage.
6.2.) Accepting liability
The development team accepted full responsibility and published a full transparency report similar to this one describing what happened in detail and the technological background of it, which is available at this link.
6.3.) Talent acquisition campaign
Subsequently, a bug bounty was launched to attract talent. 1 out of 27 highly qualified and screened applicants was able to solve the bug.
7.) 15 September 2022: Profanity vulnerability report surfaces
A report is published about a vulnerability discovered in a software called “johguse/profanity” written to generate vanity addresses (like the one generated in point 2.)).
An immediate internal audit is performed by the development team according to the following guidelines:
7.1.) Has anyone in the development team ever used this software before?
All available wallet registries have been checked. The following information is included in these lists:
- Generation timestamp
- Generated address
- Software used to generate the address
The developer assigned with the vanity address capable private key generation did not find, save and inspect the software called “johguse/profanity” but a derivative called “cenut/vanity-eth-gpu” as detailed in point 2.1.). As a result, the software called “johguse/profanity” was nowhere contained in the available maintained wallet registries.
Instead, the wallet which was found to be vulnerable on this particular day listed “cenut/vanity-eth-gpu” as the software used for its private key generation, which is unfortunately bitwise identical with the vulnerable “johguse/profanity”.
7.2.) Does any of the registries contain a wallet having a vanity address and a positive token balance?
To avoid any administrative error (since the registry is human-maintained), a secondary check was also performed to detect whether any of the available registries contained any wallet address with a positive token balance and there is any chance that it could have been generated using the vulnerable software.
Interestingly, the vulnerable wallet generated in point 2.) did not have a positive token balance, but was authorized to sign transactions. This situation is similar to someone having no assets themselves directly but being entitled to sign transaction approvals of asset transfers of others.
Unfortunately the latter (approval rights) can not be verified in an automated way similar to a standardized bulk balance check and therefore it was not included in the scope of the investigation.
Given the outcome of the two checks above, the development team was convinced that the software in question was not used by anyone in-house, and even if it had been, there was no positive balance on any of the accounts generated by such software. This led to a false sense of security.
8.) 11 October 2022: Tokens were stolen from the bridge smart contract
On this day, tokens were stolen from the bridge smart contract main account, exploiting profanity vulnerabilities.
How could it have been avoided?
To get an objective and factual picture of what could have been practically done to avoid this unfortunate event, it is important to clearly see that each individual decision should be inspected at the very time they were made and in the light of information available at that particular time, since the team did not have a time machine to make those decisions with today’s knowledge.
It took a chain of many highly unlikely events for point 8.) to occur. The following “IF” assumptions could have led to a more lucky outcome.
The starting point is the first task, which was that the bridge’s smart contract must also have a vanity address 0xAAA…AAA, similar to the token’s smart contract, in order to stand out among the possible fake bridges that could have been possibly created.
A) IF the developer would have cloned the original generator software
If the developer would have cloned the original vanity address generator software called “johguse/profanity” instead of the “cenut/vanity-eth-gpu”, then the respective wallet registry would have contained profanity as the wallet generator software used for the particular wallet, so the inspection detailed in point 7.1) would not have passed.
Why did it not happen in this way?
Quite simply because the developer did not think that there was any practical difference between the bit-identical source code of the original “johguse/profanity” and the saved “cenut/vanity-eth-gpu”, and he found the former during his search due to the reasons detailed in the end of section 2.1.).
This is a rare case that 99.999% of the time should not present any problem, since the original “johguse/profanity” and the saved “cenut/vanity-eth-gpu” both share the same bit-identical source code. The name of the software should not (and in this case did not in fact) affect the operation in any way, but unfortunately, it caused an indirect administrative problem.
Despite many suggestions, it is nonsense to claim that the developer should have downloaded “johguse/profanity” on the grounds that it was the upstream of “cenut/vanity-eth-gpu” and the latter did not contain any new code committed for the following reasons:
- In the absence of a time machine, it is not known whether the fork will be more popular in the future or the original
- Whichever version would have released (more) updates later would have had no effect whatsoever on the keys which have been already generated
B) IF the security expert would have flagged the software name as a problem
We also asked the security expert who reviewed the source code of the “cenut/vanity-eth-gpu” that was downloaded and used, whether he had checked the origin of the software to see what it was a derivative of.
He claimed that the examination had been completed and confirmed that the software under investigation contained no malicious or functional modifications to the original, so the conclusion was that there could be no functional difference at all between the two-bit identical source codes.
As described in the previous section, this was not a functional difference, but an indirect administrative error.
Why did it not happen in this way?
- The security expert stated that the software name did not affect the operation
- The security expert was not responsible for the wallet registry and had no knowledge of it
- Apart from the registry, the software name is irrelevant information in any other context
C) IF we would have carried out a collective audit after the internal assessment and verification
After 7.1.) and 7.2.) have been successfully completed, although in the absence of any objective reason, we could perhaps have conducted a collective audit of all software ever used to open a wallet, based on a “bad hunch”. This might have revealed the problem in time.
Why did it not happen in this way?
According to the information we had at the time, there was nothing to indicate that we should have feared a Profanity vulnerability, nor any guarantee that if there was any vulnerability it would be found (in time). Furthermore, it could easily be that to date, the most highly skilled people on the development team would be auditing them instead of developing the product.
D) IF a dedicated internal audit team were set up
A dedicated audit team could have been set up to compare the dozens of vulnerabilities that appear on a daily basis with all the software used by the development team with a full source code analysis (without the latter, of course, this bug would not have been detected either).
Why did it not happen in this way?
At the moment these highly qualified professionals being part of the development team focus exclusively on product development.
Furthermore, even if it were possible to find more such specialists, it would mean a significant additional investment which would not be rational at this stage of the project.
Furthermore, as discussed in section 2.2.), professionals in projects likely to have such teams and manage $5+ billion, were not able to detect this failure in time and were also proven to be exploited by this vulnerability. This confirms that such a team would not have been a guarantee to avoid the problem either, but would have only (significantly) increased the chances of avoidance.
E) IF the team did not use a vanity address at all
If the development team would not have opted to use vanity addresses for the smart contracts officially deployed by them, then the private key would have been seeded with proper randomness and the profanity vulnerability would not have been exploitable
Why did it not happen in this way?
The token launch itself has proved that malicious actors will in most cases try to create fake contract variants luring honest token transactors into using their modified malicious contract variants which mostly results in these malicious actors stealing funds in the transactions involved.
As the name suggests, vanity addresses have the benefit of easy recognisability. Generating such addresses have high resource requirements (expensive graphic cards). As a result a lot less malicious attackers can generate fake addresses looking familiar to legitimate addresses. In this particular case the goal was to make official QANplatform addresses (token and bridge) start and end with triple “A” (0xAAA…AAA) character sequence. While this does not fully prevent malicious attackers having the required resources from generating similarly looking addresses, it prevents the majority of them from doing so.
The strategy was proven to be viable, as up until today only the official contracts carry above character sequence and as thus are easily recognizable, which most likely saved a lot of funds of otherwise scammable honest actors.
Accepting responsibility
The development team accepts full responsibility for the decision to allocate 100% of the time of the highly skilled professionals who would have had any chance of detecting the vulnerability towards product development.
Furthermore, the development team accepts full responsibility for the decision of not allocating the resources of said professionals to conclude further audits related to the vulnerability described in section 7.) after the internal audits as detailed in sections 7.1.) and 7.2.) have (falsely) passed.
It is clear from the details outlined in this factsheet that such a decision would not have guaranteed that the failure could have been avoided but would have surely significantly slowed down product development. It is important to highlight that the product is neither the token nor the bridge, but a Layer1 blockchain platform bringing global breakthrough in multiple areas of the blockchain ecosystem.
According to the audit conclusions the bridge smart contract is safely operable assuming proper key storage and key usage, which is the case ever since it was deployed. The root causes of the problem were the vulnerability of the software used for key generation and administrative reasons.
Both key storage and key usage were and still are performed to the highest industry standards by the development team.
Next steps and changes
All members of the development team are convinced that such incidents being extremely unlikely to occur but happen nevertheless have additional meaning they carry along.
In light of what has happened we have clearly concluded that administrative processes are at least as important and can be as much vulnerable as purely technological aspects.
In the future, we will place considerably more emphasis on this area as well and will set up a separate technology and administrative audit departments within the organization when a particular maturity stage is reached.
Constantly updated news on the next steps
You can find out about the next steps in the recovery action in our constantly updated blog post, which can be accessed by clicking on this link.
