Networking for the Software Engineer: Network Policy

As the token operator amongst developers with a little bit of networking experience, I get a lot of requests from developers that sound a bit like this:

Hi! You know a little bit about networking, right? What does _____ mean? Can you read through this and help me understand?

Networking is an intriguing field with fantastic acronyms that rival our modern texting lexicon and works like magic. However, for the software engineer, it’s not always clear why the network matters and how it affects an application. We take it for granted that when our application sends a GET request to another application, we just receive a response. But under the hood, why does it matter? What simple terms should we know to communicate about our application networking?

This is not an overview of the OSI reference model or networking layers. It is not intended to train anyone for a network certification, help configure a switch, or outline every detail about software-defined or container networking. It will not help answer the omnipresent interview question, “What happens when you type a URL into the browser and press enter?” (see link for the answer).

What I will try to cover are:

• Networking concepts we run into when we deploy an application
• How that affects our application
• The physical, software-defined, and container technology equivalents.

When I started this story, I realized that each domain of networking needs to be split into its own. What I consider the three domains of networking are:

1. Connectivity (click for the first part): How we connect applications
2. Network Policy: How we stop applications from communicating
3. Service Discovery: How we allow applications to be easily resolved

In this story, we will learn about the common approaches to allow connectivity between authorized applications and preventing unwanted actors from accessing our application using network policy.

Why should I care?

A software engineer messaged me asking, “Why can’t my web application reach my database? It was working yesterday!” It is a often-asked question that encapsulates the software engineer’s intent, which is to allow the web application to connect to the database.

For the infrastructure engineer, this doesn’t translate much. To solve this problem, I had to ask a series of questions:

• What are the servers that host the web application?
• How is the web application accessing the database? Is there a database URL and specific port?
• What network has the load balancer that fronts the database instances?
• Is the load balancer configured correctly?
• What are the servers that host the database?
• What is the network that has the web application’s servers?
• What is the network that has the database’s servers?

The root cause? Someone reverted a network change and dropped a rule that allowed the web application servers to reach the database’s port.

This is a pretty common occurrence. The frustration introduced by a mere lack of connectivity, common language, and visibility can really affect a working relationship. Learning a little bit of the language can help narrow down the problem.

What is Network Policy?

To me, network policy is the set of rules that allows or denies connectivity between devices and services. In general, there are two models for how these rules are logically placed together in a set.

• Blacklisting: By default, allow everything. For anything we want to reject, we blacklist it. This is similar to blocking spam numbers from calling our phones.
• Whitelisting: By default, deny everything. For anything that we need to talk to or wants to talk to us, we whitelist it. This is similar to allowing someone to connect with you on a chat application.

When we think about these rules, we want to very simply say:

“If I don’t recognize the source of the request, they shouldn’t connect to my application.”

or

“My application should only be able to connect to my database and nothing else.”

This is my intent, plain language about what I intend for my network policy to look like. However, the intent often doesn’t reflect the networking domain. The networking domain is often more specific than the intent. In the situation I outlined above, we had to identify the exact network with the web application because the rules identified the network, not the actual application. Most network policy is expressed in the formulation of the network language, not necessarily the intent of the applications residing on it.

There are some Layer 7 technologies that allow us to express the intent of our network policy. This is something like “My-Database” can connect to “My-App”. However, most network policy is written in Layer 3 language, which is Internet Protocol (IP) and reflects the IP addresses and ports.

This disparity in language (intent versus network-specifics) can create some friction. We are struggling to translate the intent to the actual configuration.

For the software engineer who enjoys some Domain-Driven Design, let us think about how the network is treated versus how we speak about its hosted applications. The network is treated as one big domain. However, software is often treated as many domains.

Intent-based networking is basically applying Domain-Driven Design to the network. We start untangling the “ball of mud” network to the subdomains of the business logic it serves.

We have not quite achieved intent-based networking, so it still helps to know about the network policy terms we might encounter. For the remainder of this discussion, we’ll refer to:

• Network-domain specific language as rules expressed through IP address and ports.
• Intent-based language as rules expressed through intent.

The technical implementation tends to be distributed across many types of technologies.

Network Policy Technologies

There are many devices and technologies in the datacenter that contain rules about network policy. When troubleshooting, we encounter several types of these rules and end up switching between devices to find them!

Physical Devices

These are usually firewalls or access lists on network devices, like switches or routers. Most physical devices uses network domain-specific language like IP addresses and ports to express network policy.

Firewalls are devices that block certain packets, depending on the rules we set. We tend to use firewalls for two purposes:

• Blocking packets from malicious actors
• Preventing interactions between two internal services, applications, or environments. For example, preventing traffic from being passed from a development environment to production.

Network Access Control Lists (NACLs) are rules often set on routers and switches to control network access. They mostly control network policy within the datacenter. These can also be used to prevent interactions between services, applications, or environments.

Software-Defined Network Policy

Software-Defined Network Policy mixes intent-based and network-domain specific language. We usually have the option of declaring “groups” of application instances that are allowed to connect or declaring a specific IP address range. Sometimes, declaring network policy rules by the groups of application instances is considered microsegmentation.

If we have some kind of software-defined networking in our own datacenter, we can implement rules on our controller to drop or allow traffic. Similarly, most public clouds (Amazon Web Services, Google Cloud Platform, or Microsoft Azure) use software-defined networking to provide users a network on-demand. We often hear about security groups or policy groups. These are logical constructs that provide finer-grain control over traffic management by applying metadata to the components we’ve grouped together, allowing or denying traffic based on our intent.

There are also virtual firewalls that can address these concerns. These include Web Application Firewalls (WAFs) on public clouds. Similar to the physical firewalls, these generally use network-domain specific language. There are a few virtual firewalls that use intent-based.

Another interesting software-defined network policy implementation is iptables. This uses the kernel’s netfilter module to selectively filter or forward traffic on a Linux host. Actually, Docker uses this to filter and direct packets when we deploy a container onto the default Linux bridge. It is a bit like a firewall for each host.

Container Network Policy

Container orchestrators tend to have their own method of enforcing network policy. For example, Kubernetes uses NetworkPolicy to control pod-to-pod traffic. This tends to control Layer 3 traffic, the traffic that is being transmitted in the IP address space. Additional technologies like Envoy proxy that controls traffic between services using additional metadata on Layer 7.

Containers also have their fair share of networking options that plug into the container engine and help manage container network interfaces. We won’t cover all of them but many of these container networking technologies also have network policy features. For example, Calico has network policy rules to drop traffic to and from containers.

Service mesh can also provide traffic management for situations with many services. Service meshes are most Layer 7 focused, which means that it routes traffic based on the application-level intent. For example, Istio uses Envoy proxy to facilitate traffic management. Its intent it not necessarily to drop traffic but it does provide quite a bit of traffic management that is useful for even more fine-grain control. Perhaps we would call this nanosegmentation?

Summary

All of these technologies will overlap with certain layers of the OSI model mentioned above.

Keep in mind, these are approximations. Depending on the technology, we might find certain situations where it extends to to other layers.

In Summary

It seems so simple to state that we want to allow something to talk to another or deny the connection with something else. While we spent more time covering connectivity, in some ways, it is the least complex of the set of networking problems. Network policy is a ball of mud on its own — interwoven with many rules on many layers, often configured by different teams with different language. It is difficult to get a single pane of glass outlining every rule as a result. As we continue to work with containers, we are building tools that allow network policy to expressed as intent.

There is still one question left to ask, however. How do we find our service in the first place? When we type www.medium.com how does it get to our service? The answer? Service discovery.