Securing RubyGems with TUF, Part 1

Applying The Update Framework (TUF) to RubyGems to secure it against nefarious activity.

Over the last week, a small team¹ here at Square has been working on integrating The Update Framework (TUF) with RubyGems to help protect the community from compromises like the one that happened earlier this year.

In this series of blog posts, I aim to explain the fundamental concepts of TUF and how they apply to RubyGems. This is not a rigourous explanation or justification of the security properties of TUF (refer to the specification for that). Instead, it will equip you with the mental models needed to go exploring and learn yourself. I have simplified many of the details for sake of explanation.

Fundamentally, RubyGems is a file server. To download a gem, I first fetch the index (/latest_specs.4.8.gz) to determine the latest version. From there, I construct a URL to download the actual gem (/gems/somegem-0.0.1.gem). If a malicious user gains access to the server, they can change the contents of these files to serve malicious code. TUF aims to defend against this attack and more.

Securing A File

I’ll start with a high-level overview of the system. The details are explained further down, so I recommend not spending too much time on this the first time around, and then coming back and reading it in detail after you’ve read the rest of the post.

TUF adds metadata files in parallel to the existing rubygems files. This metadata stores digests of the files which can be used to verify the files’ contents. The trick is to ensure that this metadata can be securely delivered to the client.

To do this, TUF divides the metadata into different files called roles, and each role is asymmetrically signed by one of three types of keys: an offline key, an online key, or a developer key. Each key has a private and a public component: the private part is used to sign a role, and the public part is used to verify that signature.

The private part of the offline key is stored off the internet (“air-gapped”) by the RubyGems maintainers². If the RubyGems server is compromised, the offline key is still safe. The private online key is stored on the RubyGems server, and so is not safe in the event of a compromise. (The need for this unsafe online key is covered below.) The private developer keys are stored securely by developers, and I’ll cover how they fit into the picture in part 2.

The public parts of all keys are published in the metadata files. In addition, the public offline key is distributed with RubyGems. This is required to bootstrap the process; we need a known good key that is not told to us by the server. Otherwise a compromised server could just serve up a bad key and we wouldn’t know the difference!

Let’s try serving up just a single file with a simplified version of TUF. The file tree looks like this:

.
├── metadata
│   └── targets.txt
└── my-super-file.txt

First, the client requests /metadata/targets.txt, which contains the following:

{
  "signature": { "keyid": "offline", "sig": "196b99cd975c8ab9dc70" },
  "signed": {
    "files": {
      "my-super-file.txt": "cb18ca7e4084820d53dc444b97c253b3"
    }
  }
}

This document was created by a RubyGems maintainer, signed using the offline key.

Using the public offline key we have locally (distributed with RubyGems), we can verify that the signature is valid for the signed part of this document. This gives us confidence that someone with access to the offline key created this document. Now we can request my-super-file.txt and validate its digest against the one provided in this file. Even if an attacker gains access to the RubyGems server they cannot access the offline key since it is not stored on the server. While they can shut down the server, they cannot serve up malicious content.

Online Operation

That system is really secure, but consider what has to happen to add a new file. A maintainer needs to manually sign the new metadata file on their non-internet connected computer, then transfer that to the rubygems server. What a chore! Clearly this does not work for a system like RubyGems, where new files are being added all the time by many different people.

To get around this we could use the online key instead of the offline one, which is available on the RubyGems server and so can be used to automatically sign new metadata files whenever a new gem is pushed. The problem is that if the server is compromised, the attacker can replace the online key with their own version and use that to sign malicious versions of gems. We are not totally back to square one though; using an online key in this way protects us from any man-in-the-middle attacks, such as the compromise of a mirror.

Mix And Match

We seem to be at an impasse. An offline key is the most secure, but requires human intervention to apply. An online key can be used by robots, but is more easily compromised. Can we get the benefits of both?

We can! TUF supports a concept called delegation. Rather than providing a list of files, a target role can instead provide a list of other targets to go and reference for files.

.
├── metadata
│   ├── targets
│   │   ├── recent.txt
│   │   └── verified.txt
│   └── targets.txt
├── my-new-file.txt
└── my-super-file.txt

# targets.txt
{
  "signature": { "keyid": "offline", "sig": "be49f3a66314cbaf" },
  "signed": {
    "public_keys": { "online": "7b1fd6094c6b87c196f1ff423527da38" },
    "delegations": [
      { "name": "verified", "public_key": "offline" },
      { "name": "recent", "public_key": "online" }
    ]
  }
}

# targets/verified.txt
{
  "signature": { "keyid": "offline", "sig": "196b99cd975c8ab9" },
  "signed": {
    "files": {
      "my-super-file.txt": "cb18ca7e4084820d53dc444b97c253b3"
    }
  }
}

# targets/recent.txt
{
  "signature": { "keyid": "online", "sig": "305781c28a40f8639" },
  "signed": {
    "files": {
      "my-new-file.txt": "a5cde2fafbeb6603096e064def328421"
    }
  }
}

Now to find my-new-file.txt, a client fetches and verifies targets.txt, then starts iterating through its list of delegations. First it tries targets/verified.txt, which it verifies with the offline key, but it does not contain the file it is looking for. recent.txt is next on the list, which is verified with the online key, and does contain the digest for my-new-file.txt.

On the server, any new files can automatically be placed in the recent role. At regular intervals, a person promotes all files in recent to verified by moving all the entries into verified.txt and re-signing it with the offline key.

If the server is compromised, all files in recent.txt are vulnerable to tampering; files that have been promoted to verified.txt are safe.

Using this mixed strategy we get the benefit of offline-level security for old files, while still enabling new files to be handled automatically. Pretty neat!

In the next installment, I’ll cover how TUF enables developers to securely update their own gems to help get RubyGems maintainers out of the loop.

¹ Myself, Tony, Neal, Justine, Charlie and Jim. Shoutout also to Justin Cappos and his students at NYU-Poly for providing examples for us to work off, answering our questions, and reviewing our code!
² Actually there are multiple offline keys. Each maintainer can have their own. A threshold number (usually three, but configurable) is required to adequately “sign” anything. The same arrangement is also true for developer keys. I ignore this for simplicity in the main text, but it is an important feature of TUF.

Xavier Shay (@xshay) | Twitter
The latest Tweets from Xavier Shay (@xshay). Engineering Manager at Square. Vegan. Running is the best. San Franciscotwitter.com