How we do security assessments at Shift

Kaspar Etter
Dec 2, 2019 · 14 min read

Over the last two years, security researchers reported to us several vulnerabilities in the BitBox01, and we found some internally. Through this process we gained some experience with assessing their severity. With our recent responsible disclosures to our competitors, we realized that the industry of hardware wallets lacks a common standard for severity assessments. This blog post outlines how we evaluate vulnerabilities at Shift. By publishing our framework, we hope to get feedback and to move the industry forward. We are aware that our classification isn’t perfect, but we haven’t yet found a more suitable model.

Image for post
Image for post

Threat model of hardware wallets

Before we dive into potential vulnerabilities, let’s first discuss why a hardware wallet is still the best means to hold your cryptocurrencies despite reported vulnerabilities. Devices like computers and mobile phones, which could be used to store a wallet’s private keys, run highly complex general purpose operating systems with millions of lines of code and have weak security domains. Unlike those devices, hardware wallets typically only run dedicated firmware without any user-provided code and are built for a single purpose: keeping your private keys private. This reduces the attack surface a lot, which means attackers have fewer options to steal your private keys and thus your funds. In order to be secure, you should always trust the screen of your hardware wallet over the information displayed on your computer and verify carefully what is displayed on your hardware wallet. The manufacturer can add exceptions to this rule, for example when your computer warns you about a failed device authenticity check (i.e. you may have a potentially counterfeit device). Protection against malware on your computer is the bare minimum a hardware wallet should provide in terms of security. Most hardware wallets also aim to protect your assets from theft when your device is stolen by encrypting sensitive information and requiring a PIN or password to unlock the device.

While the above might seem trivial, there are many situations where the covered security threats are not clearly defined. This is why it’s important that hardware wallet companies publish a threat model, where they explain what they protect you against and — more importantly — what not. Here are some examples to illustrate why the scope of attacks and countermeasures is not trivial and needs to be made explicit: What are the mechanisms in place to prevent tampering in the supply chain before the device reaches you? Are evil maid attacks in scope or out-of-scope? How can you be sure that the firmware binary you install on your device is the one for which the code is published and which everyone else is receiving? Or is the company being forced to sign a malicious firmware just for you? Can you trust the receive address for a multisig account that is displayed on your hardware wallet? Are you still secure if the cosigner of a multisig setup is malicious or compromised? We will come back to the last two questions in a future blog post. You find the answers to all the other questions regarding the BitBox02 in our threat model. The only other threat model we are aware of is the page on security threats by Trezor. Ledger lists which vulnerabilities are in-scope and which are out-of-scope in their bug bounty program.

Severity = impact * scalability

The risk of a particular vulnerability for users, and thus its severity, is determined by multiplying the impact by the likelihood of the exploit. In general, the easier it is to perform an attack at scale, the more likely it is that you are affected. In particular, since it is easier to distribute malware than to steal a large number of used devices, you are more likely to be a victim of software-based attacks, also known as remote attacks, than hardware-based attacks. However, this depends a lot on your personal situation. If you are a high-profile individual or bring your hardware wallet to cryptocurrency meetups and forget it there, then the situation can be different for you. For simplicity, we assume here that the likelihood of you being attacked based on a particular vulnerability scales proportionally with the overall number of users that the given attack could target simultaneously.

Impacts of vulnerabilities

By finding and exploiting a vulnerability, an attacker can achieve one of the following outcomes, listed in order of decreasing severity:

  • Seed extraction/injection: The worst outcome is when the attacker learns the seed of your wallet, from which all keys are derived. An attacker can steal your funds immediately or wait until you accumulate more coins and tokens; then withdraw all your funds when you least expect it. This delay from the attack to the time of exploitation makes it harder to link the attacker to the attack and find the vulnerability. Furthermore, unlike in the case of stealing funds, where the problem is typically in the transaction signing implementation of a particular coin, learning the seed compromises all your funds and potentially other features like U2F/SSH authentication or email signing/decryption. An attacker knows the seed if they manage either to extract it from the hardware wallet or to inject it into the hardware wallet when the user sets up the wallet.

Scalability of vulnerabilities

Vulnerabilities can differ drastically with regards to how easy they are to exploit at scale. As mentioned above, the more users that can be attacked, the more likely you are one of them. Exploits where the attacker doesn’t have to be physically present are easier to scale. The following attack vectors are listed in order of decreasing scalability and thus decreasing severity.

Remote attacks (performed with software)

In the hardware wallet industry, attacks that can be performed from the computer to which the hardware wallet is connected are considered to be remote: no physical access to the device is required as the exploit happens through the API of the hardware wallet. How you get malicious software on your computer is outside the scope of this article, but, as noted in the threat model above, this is the main attack vector hardware wallets should protect you from. Remote attacks are really bad as they can scale quickly. For example, the download server of the manufacturer’s wallet software can get compromised and most users would likely install a malicious app as they don’t verify the signatures on the downloaded binaries. The following distinctions depend on the logic of the hardware wallet and do not apply to all products equally.

  • Without unlocking: The most severe remote attack is when the device does not even have to be unlocked in order to be exploitable. While a user typically connects a hardware wallet to a computer in order to unlock and use it, an exploit without unlocking might also be possible via websites that you visit through the U2F interface, which quite a few hardware wallets support. Additionally, depending on the nature of the vulnerability and the architecture of the hardware wallet, it could put further, unlocked wallets on the device at risk (e.g. the ones you created for plausible deniability). Exploits without unlocking are also very useful for attackers who gain physical possession of your hardware wallet.

Supply chain attacks

A hardware wallet or its accessories like the USB cable can be tampered with before it reaches you. This kind of attacks are significantly costlier to perform than software-based remote attacks, since it involves physical modification or replacement of components. Depending on where in the supply chain the attack is performed, such attacks could also scale quite easily. However, as sophisticated supply chain attacks are hard to protect against, they are often not covered in the threat model of hardware wallets.

Local attacks (performed with hardware)

If, in order to exploit a vulnerability, the attacker needs physical access to the device after the user has already created a wallet on it, then we talk about a local attack. Please note that the boundary between local attacks and supply chain attacks is vague, as many local attacks can be transformed into supply chain attacks with some effort.

  • Non-invasive without returning to user: If a found or stolen device doesn’t have to be cracked open and doesn’t have to be returned to the user in order to exploit a vulnerability, then such an attack is easier to scale than the other local attacks as the device just has to be plugged in. Once built, no knowledge about embedded systems is required and thus the equipment to perform the attack can become attractive on the black market.

Other aspects to address

The above scales for the impact and scalability of vulnerabilities are sometimes too simplistic. Often there are additional aspects to consider on a case-by-case basis. The following questions are examples of aspects that should also be addressed by the vendor if applicable when informing its users about a vulnerability:

  • Which coins are affected by the vulnerability?

Severity assessment

The adjectives typically used to label the severity of a vulnerability are in order of decreasing severity: critical, high, moderate, and low. There can be reasonable disagreement about how to apply these labels to vulnerabilities, or in other words, how to quantify the impact and scalability of attacks. Given the considerations in the previous sections, we define them along the following lines:

  • Critical: All remote attacks up to and including locking the user’s funds for ransom because they can be performed by a financially motivated attacker at scale with zero marginal costs.

Since many vulnerabilities have aspects that make them “special” as highlighted by the questions in the previous section, we reserve the right to deviate from these rules in our bug bounty program at our sole discretion. Furthermore, it is often not clear what the impact of a potential vulnerability is, especially in the case of memory issues such as leaking some bytes.

Image for post
Image for post
Our security assessment table. * means that the severity depends on whether an advertised mechanism can be circumvented. Please note that additional factors influence the severity evaluation of a particular vulnerability.

Let us know what you think about this framework. We expect this framework to evolve over time and hope it helps drive the conversation forward about hardware wallet security.


I want to thank Jad Zeidan for developing and maintaining our original scoring system, and Sebastian Küng and Douglas Bakkum for giving valuable inputs to this blog post.


Swiss made hardware wallet BitBox02. Get yours:

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store