Netflix Culture Meets Product Security

Netflix Culture

A lot has been written on the Netflix culture. There’s the official culture deck, several employee stories, journalist articles, and probably many side conversations throughout Silicon Valley and beyond. The tldr is that, at Netflix, we strive to hire great employees and then trust them to do their job well. So we are light on policy, and instead just ask people to work in the best interest of Netflix. In practice, the degree to which we push this philosophy into our everyday work is striking. If you haven’t seen it, read the Netflix culture deck now. The rest of this article will make more sense with that context in your head.

When I talk with others in the security community, a common question is “How does security work with Netflix’s culture?” Understanding this really gets to the core of our culture. To be honest, I’m quite fond of how this works. We keep Netflix secure. And we keep developers moving quickly.

Product Security at Netflix

At a high level, I’ve seen two models for handling product security at tech companies:

1. Security is the responsibility of each individual product team
2. Security is the responsibility of the security team

Netflix operates under the first model. The security team provides the services, libraries, tools, and expertise to help other engineers secure their work. But it is the responsibility of each team to make use of these resources. In practice, this means that a member of the product security team may file a bug report, but they are unlikely to create the corresponding pull request.

Under this model, security works best when teams know that they can (and should!) leverage the resources on the security team. Many companies would use process and gates to force this interaction. For example, this could mean requiring security sign off before pushing new code into production. At Netflix, we shy away from such approaches. Instead, we rely on the judgement of our engineers, old fashioned relationship building, and some pretty slick automation. By focusing on our relationships with other teams, we build trust and ensure that everyone understands Netflix’s security priorities. This approach is likely worth an entire blog post to discuss in depth. But the automation parts are worth a brief discussion here as it is central to our approach.

I mentioned above that some companies enforce a process that requires security approval for all new projects. However, what if the security team was omniscient? What if the security team knows about all projects, how they are deployed, who they communicate with, what data they handle, and more? What if the security team knew all of this in real time? In this mythical world, you could skip any approval process and automate a security check for all new projects as they are created. That security review could happen transparently and in parallel to the project work so as to not slow down the engineers. Wouldn’t that be amazing? Sadly, we aren’t omniscient. On the other hand, we do have the next best thing. We deploy into the cloud. And this means that we have strong audit trails that allow us to collect this kind of information and act on it. We use tools like Scumblr to make this happen.

In summary, by thinking outside of the box we are able to better align our product security approach with the Netflix culture. Next, let’s dig a little deeper on specific aspects of the Netflix culture and how they have shaped how we operate as a security team.

Culture Meets Security

Stunning Colleagues

Have you ever worked with a person who just blew you away with their talent, maturity, passion, selflessness, and integrity? These are the people that we remember for years because they left such an impression on us. For most people, there are only a few coworkers in each of their jobs that fit this description. At Netflix, we work hard to fill the office with these people. Take a moment to think about that. The work dynamic is fundamentally different. People default to assuming their coworkers are experts in their fields. And quality, impactful work happens at a blistering pace.

If the product security team can’t rise to this level of excellence, then we would quickly damage our relationships with other teams. As a result, this would ultimately lead to not being able to positively impact security. To avoid this, we keep a very high bar for members of the security team. This doesn’t necessarily mean that everyone must have decades of experience. Instead, it means that each person is a stunning colleague for where they are at in their career and what their role is on the team.

Hiring stunning colleagues is hard in general. Doing this within the security space is even harder. So this requires patience and creative approaches to recruiting.

Freedom & Responsibility

As people grow older, freedom and responsibility tend to grow together. Starting at a young age, kids are able to increasingly take responsibility for themselves. This responsibility complements additional freedoms. For example, most kids wouldn’t be permitted to walk home from school alone until they were responsible enough to do it at the right time and safely.

At Netflix we aim to hire “fully formed adults”. One reason is that these are people that can handle significant freedom responsibly. We believe that this freedom allows Netflix employees to successfully navigate an ever changing world much better than lots of rules.

In practice, this culture of freedom and responsibility impacts our security program in a couple of ways. First, engineers at Netflix work in a variety of different tech stacks (different programming languages, libraries, micro-service dependencies, etc). This increases the load on our security team. It also encourages us to solve problems creatively. For example, if we can provide an authentication solution that works across any tech stack, it will be more useful than a Java-only solution.

Freedom and responsibility also provides a backdrop for defining how Netflix’s product security teams interact with other teams. Each team is responsible for the security of what they create. And the security team is here to provide the context (i.e., relevant information) needed for these teams to make good decisions. This parallels how we think about other specialty areas as well. For example, everyone needs to think about the performance of their code to some extent. But when the performance considerations require a specialized expert, people can reach out to a member our our performance team for help.

Context, not Control

At Netflix, we use the word “context” a lot. After hiring stunning colleagues it would be a waste to diminish that talent. We want people to bias for action and to feel comfortable leading large initiatives without top-down approval. So rather than creating processes and social norms where management approves everything, we empower employees and ensure they have the relevant information to use their good judgement to make decisions. Context is this relevant information.

In many companies the security team inserts themselves into lots of business processes. This could mean requiring security approval before pushing code into production. Or it could mean requiring security signoff on foreign travel. All of these security requirements are gates that slow things down. These situations are how security teams start to be known as the team that blocks everything. Over time, people stop wanting to talk to these security teams, find ways to creatively avoid them, and the security stance of the business gets worse.

At Netflix, we work hard to not control other teams. This means we are careful with our use of language. For example, “recommend” is preferred over “required”. While it may be counter-intuitive to some, we have found that this improves security overall by creating an environment where people want to partner with our security teams.

Does It Work?

You may be thinking that this all sounds great, but wondering if it really works. The short answer is yes, it works. When comparing our security program to others in the tech industry, we are very pleased with how we are doing. Digging deeper on how we think about our security maturity and how we measure for success is a topic for another day.

In short, I think the model described above works at Netflix because this is our culture. These themes of “Freedom & Responsibility” and “Context, not control” are the path to success for all teams at Netflix.