From the Ground Up: Building Product Security at Uber

Because software has inherent vulnerabilities, smart security teams build protections inside and outside their code to help prevent exploits. The goal is not only to limit the impact of an unknown vulnerability, but to prevent future vulnerabilities from ever being written into code. To do this effectively, security must be present and involved throughout the entire development process. This is the purpose of the product security team at Uber.

We see security as a subset of quality. Alongside performance, usability, maintainability, correctness and cleanliness, security should be a quantitative consideration when writing software. At Uber, we have found success by following three key principles.

Solve for Yes

Culture is an important tool for improving security. It’s easy for security teams to simply say “no.” It’s much harder to figure out how to safely get to yes, but this is the job of a good product security team. At Uber, we challenge historical assumptions about what can be secured. Establishing a reputation of saying “no” with your product and engineering colleagues kills a lot of potentially work and weakens your influence with these teams.

We committed to a culture where other teams aren’t afraid of security. We don’t allow shaming. Instead, we work hard to make something both possible and safe. This attracts a lot of inbound work for our product security team, builds strong relationships with product engineers, and forces us to think critically about short and long-term challenges — — and new ways to solve them.

As a result of these efforts, the product security team Uber is a welcomed advocate throughout the development process. Before a single line of code is written we investigate the impact new builds will have on current architecture and ensure it’s not inherently impossible to secure.

During this conceptual phase, we ask questions such as:

• What data will be stored and/or used by a new service, feature, or product?
• Where is input validation done? How and where is authorization enforced?
• Is the software directly accessible to external parties, e.g. vendors, partners, public, etc.? If so, how will they access it and what controls are in place?

We work to identify hard tradeoffs that might weaken our security posture and help product teams make safer decisions. Catching issues at this stage means potential flaws are never written to begin with, and solidifies the partnership between security and engineering.

Follow the Bugs

Each bug is a lesson, a story of mistakes and a valuable data point in finding potentially larger problems. It is our job as the product security team to follow bugs back to their root and understand the deeper cause.

Software is written in stages and our security team gets involved in each one because different types of vulnerabilities can be introduced during different stages. As a result, different security strategies and processes are used during each stage to identify as many bugs as we can before code ships into production. The early we find a bug, the easier it is to fix it with minimal impact on subsequent stages. This is good for both security, and cost.

Phases of software development at Uber and some of the corresponding security tools for detecting bugs. Our goal is to identify potential security issues in the earliest stage possible.

Here’s a brief outline of what our security efforts typically look like during each stage:

• Design — We guide development away from insecure decisions through reviewing the design. This is the best stage to snuff out a bug before it ever becomes code.
• Build — Code exists but it is in flux. Finding & fixing individual mistakes in the code through advice and awareness. The product security team serves as consultants to product teams, answering questions like “is this secure?” or “how should I write this?” and proactive outreach.
• Ship — A thorough review by the product security team, if appropriate before launch. Our product security team holds itself to an SLA and we work hard to keep launches on track.
• Live — Software spends most of its lifetime in this stage, so our security efforts don’t stop at launch. In addition to ongoing internal tests, we also invite external researchers to audit our code in production through the Uber bug bounty program.

Every bug is a lesson — an indicator of inexperience or misunderstanding of how something works. This is true of the engineers writing code and us on the security team looking for bugs. Over time, these bugs provide a long tail of tactics to help improve our code, which we can enforce on new projects.

At Uber, we shamelessly follow leads from sources that routinely produce bugs — a specific part of the codebase, abstraction, or team. This helps us formalize bug patterns to catch them before they go live. Sometimes it’s a grep-based checkin rule, static analysis rule, or a specific individual needing more training. Either way, by working back from each bug, we accumulate customized tools and processes tailored specifically to the problematic characteristics of our code.

Measure & Prove Your Worth

Security can be difficult to quantify because you cannot measure an unknown. The most common perception, even within our own industry, is that security is expensive insurance and a cost center. In terms of measurement, this approach leads security professionals to ask abstract questions like, is one critical bug worth 20 low priority bugs, or if we look harder for bugs and find more is that a good thing?

At Uber, we decided to look at measurement differently and track the effort required to prevent bugs from being created in the first place. We believe security is more than stopping bad things from happening — it should be valued for its ability to unlock new types of business by providing a trusted foundation. The ultimate goal is to contribute to both the technical and financial health of the company.

We know bugs are more expensive to fix the longer they go through the development process before they’re found, and so, identifying security issues in earlier stages of the process benefits both cost and safety. This is one way we prove our worth to the business — demonstrating how our security team moves these fixes “up the chain” to earlier stages of development, ultimately preventing many bugs from ever being created.

Even bugs found in live code lead to improvements earlier in the process, maybe a static analysis rule, better education, or a library. Additionally, our bug bounty program compliments internal discovery of bugs in the wild. All of these signals can be used in reporting our value to the business by tying directly to the impact on development.

How it Works in Practice

Here’s an example of how the approach above plays out. Jinja2 RCE was reported by a security researcher through our bug bounty program in April 2016. After fixing the bug, we built a code commit rule that looks for any patterns like this in our codebase and alerts the product security team for manual review. This effectively moved detection of this bug earlier in the development process. We also modified the jinja2 engine to take out the ability to run python code inside an expression. Now, even if a developer mistakenly writes this bug again, it is defanged.

Security by Default

As more and more bugs move toward the best end of the spectrum, we prove our value to the business in a way they understand — — development time, risk, and cost. It also puts us in a position of strength for conversations about resources, tooling, and product strategy.

We settled on these guiding principles to help keep Uber’s code clean and secure. They also let the builders at our company build new and better experiences for users while achieving increasingly more security by default.