Docker’s Head of Security David Lawrence: on TUF, Notary, and the importance of software security

Last Fall the Cloud Native Computing Foundation (CNCF) added two open-source security projects to its roster, Notary and The Update Framework (TUF). Both projects improve software integrity and provenance and by doing so are expected to help accelerate cloud adoption more broadly. Originally developed by Docker, Notary provides high levels of trust over digital content using strong cryptographic signatures. Notary is the most widely adopted implementation of the TUF specification, which helps secure both new and existing software systems.

We were thrilled to sit down recently with David Lawrence, head of Docker’s security team and the principal custodian of Notary, to discuss his background, TUF’s impact, and the evolution of Notary.

Can you share your background with us?

For a start I’m not from California. I have a lot of family up in the Boston area and knew that when I graduated I wanted to move to California, live in Silicon Valley, and work at a startup. I got lucky because the CTO of the first company I worked for graduated from my college, and her company had an intern program. I started at the business as an intern and then joined full-time. It was a tactical move to get me to California but wasn’t the technology or industry area I wanted to build a career in, so I moved on after a year. I joined a stereotypical Silicon Valley startup working out of a house with the conference table in the garage. That company was Bitcasa, which built an encrypted cloud storage platform. Joining a startup at such an early stage let me contribute in a lot of ways, touch a lot of technologies, and learn a huge amount. Over my time there I was able to grow the engineering org and ended up leading the platform team, building the core backend storage tech.

What led you to join Docker?

I decided that it was time to move on and got connected to Docker through networking with VCs. Having spent three and a half years working on encrypted storage, I specifically joined Docker to build a trust solution that enabled someone creating content and deploying it onto servers to get a high degree of trust that it hadn’t been tampered with in between the build and arrival on the server.

At that time, what options existed to build a trust solution?

There were really only two camps of thought at that time. First is GNU Privacy Guard (GPG) signing which has been used for decades but has issues, particularly in the context of software. Some of the properties that make GPG perfect for signing an email make it challenging in software, a case in point being you can’t go back and tell the user not to trust the content anymore. In software we need to have a way to say that the software isn’t valid. We have to be able to say that we found vulnerabilities or bugs in the software. GPG doesn’t let you do that. You can’t go back and revoke the signature. Around the same time period that I was transitioning to Docker, there was a project called TUF that came out of New York University based on an updater built for the Tor project.

Tell us about TUF?

The group behind TUF is led by Professor Justin Cappos at the NYU Tandon School of Engineering. Once we started on the trust solution project we identified that TUF was the direction we wanted to go. Justin and team were great in an advisory role as we were able to discuss details of the specification and ensure we implemented it correctly. I’ve worked closely with him and his team, and today Notary is the most productized implementation of the TUF specification.

In my opinion GPG is a primitive. It gives you a way to sign something but little else. Everybody starts adding additional pieces on top of it to create the properties that they want. Nobody has formalized that set of properties for GPG. You can’t go and look up a signing specification. You really must go through the changelog to see changes made for particular attacks.

TUF did an analysis of the major package managers like rpm, Debian, Python, and looked at the attacks that had been carried out against those package managers and the protections that had already been implemented. Through this work, they came up with a well-defined specification. The entire spec is readable in an hour or two. They took a platform approach to make sure a well-defined collection of security guarantees are met. We chose TUF because we really liked this approach in addition to the high level of quality.

Additionally, we had the option to use TUF or build something ourselves. We really liked that TUF was designed by security experts and had been peer reviewed. TUF accelerated our R&D and allowed us to get a strong solution to market faster so people could be better protected sooner.

What are some strengths of TUF?

TUF is wonderful at ensuring the integrity of updates and provenance of where they came from. Most systems focus on integrity. Integrity ensures that you receive what you asked for and it hasn’t been tampered with. Integrity doesn’t let you know if the person you think produced the code actually produced it or if it was contextually correct for you. This is where provenance steps in. GPG requires significant extra work to add provenance while TUF has the concept built into the framework.

With TUF you sign a collection of content that has a context, and you can build your process into the collection, making content only valid if it has the correct set of signatures. For example, you can define structures in the framework that would make content valid for QA if it has two signatures, but to go to production it needs three, and one of those three must be a member of the QA team. With GPG if you want those guarantees you have to build them on top. It doesn’t come with it.

What is Notary?

Notary is a productionized and purist implementation of the TUF specification. It is written in Go and comprises a server and a client for running and interacting with trusted collections. Today Notary is most widely deployed through Docker Content Trust where it provides robust security for container image updates. It helps address software integrity and provenance to create a more secure software supply chain.

We’ve seen it picked up by others like Kolide, which is a commercialization of osquery. Without coming to us, Kolide was able to pick up and use Notary, which I consider a good indication of its ease of integration. They use it during the update process to sign their updated packages and distribute them in a secure manner. If someone wants a signing solution Notary is a great option.

What’s the current state of the market?

A lot of the market isn’t ready for key management. If we could solve key management, we would solve a lot of security problems. We have reasonable solutions for public key management, particularly in controlled environments like an enterprise. However, people are working on many devices now so private key management is getting harder. If I want to sit down at my laptop and publish a piece of software I need my key available to sign that software. But when I go to my desktop I either need the same key available or need to have taken some positive action that links the keys together to demonstrate that they both represent the same person.

There are some systems that try to address this like GitHub. When I want to add a key to GitHub I go to the web UI and add the key to my account. Unfortunately, the identify of that key is only as good as the user authentication on GitHub. If someone cracks the user authentication they can add a new key.

In Notary and TUF if I add you as a collaborator on the repository I need to get your public key and sign it into the repository along with rules for specific items you are permitted to add to the repository. The challenges are how do you get your public key to me. Typically, you would do that through email, which is itself not an entirely secure solution. Another issue is if you lose your key or need to change it. And here Notary and TUF provide clear and straightforward recovery paths to rapidly revoke or change the keys in use.

Why haven’t we seen a breakout key management company?

There have been companies that have gone after key management. The first hurdle, which will never go away, is that humans will turn off security that gets in their way. For any form of key management, a high priority goal must be to not reduce security to a user credential or a token or something that you’ve deployed onto an online server that someone could capture and use to go authenticate against your key management server. This is a problem with Hardware Security Modules (HSMs). They are great, but you still need to control access to the HSM itself. The benefit of the HSM is that the attacker can’t lift the private key material off the device. So as soon as you cut off access to the HSM they lose access to the key. They can’t keep using it in some other context that might continue to compromise you.

Closing out it would be great to hear your vision for the future of TUF and Notary?

We donated Notary and TUF to the CNCF together because we thought it would be a stronger submission. TUF is the specification and Notary is the leading implementation. These were the first security projects donated to CNCF. We believe TUF will see more adoption in IoT. We are starting to see this with Uptane, an implementation for the automotive industry.

I think Notary is going to be most successful as a library that is integrated into other solutions that hide a lot of its complexities. We are starting to do this in Docker already by building the tools that take Notary’s opinionation into account. Notary has been integrated into the Docker tools and commands have been streamlined. I think users want an opinionated implementation that they experience through tooling.

Any final thoughts?

I want to thank the community for their engagement, and they should look forward to an update soon!