Open in app

Sign in

Write

Sign in

Namespaces in Go - Network

Ed King
8 min readJan 9, 2017

--

In the previous article we saw how to make use of PivotRoot and the Mount namespace to swap in a new root filesystem for ns-process. With that change in place, ns-process is starting to look and feel an awful lot like any other container. Sure, it only runs a single /bin/sh process at the moment, but it does have a number of extremely cool features:

  1. Can be run as a non-root user thanks to the User namespace
  2. Can choose a root filesystem to run in thanks to the Mount namespace
  3. Cannot see any of the host’s processes thanks to the PID namespace

That’s pretty impressive! But there’s still a piece of vital functionality missing - networking. At the moment, ns-process doesn’t have any network connectivity!

💁 The following has been tested on Ubuntu 16.04 Xenial with Go 1.7.1

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1$ go build
$ ./ns-process
-[ns-process]- # ifconfig
-[ns-process]- # route
Kernel IP routing table
Destination     Gateway         Genmask         ...   Use Iface
-[ns-process]- # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network is unreachable

That’s slightly less impressive… The reason for this lack of connectivity is due to the fact that ns-process clones a new Network namespace, the very purpose of which is to isolate all network-related resources (IPs, ports, interfaces, etc.).

In this article we will set about configuring the new Network namespace such that it ends up with an interface and a routable IP address.

🌐 A quick lesson in networking

If we are to have any hope of adding network connectivity to ns-process, a solid understanding of the Network namespace is going to be essential. To that end, I highly recommend you read through Introducing Linux Network Namespaces. The knowledge and ideas presented in that article will form the basis for the Network namespace configuration in ns-process. To briefly summarise, here’s what we’ll need to do:

  1. Create a bridge device in the host’s Network namespace
  2. Create a veth pair
  3. Attach one side of the pair to the bridge
  4. Place the other side of the pair in ns-process's Network namespace
  5. Ensure all traffic originating in the namespaced process gets routed via the veth

The general idea is to establish a connection between ns-process's Network namespace and the host’s Network namespace. Visually this looks a little something like this:

This is actually a fair amount of work! And it’s made complicated by the fact that setup and configuration needs to occur in two different Network namespaces. There’s also a further complexity in that the network setup requires root privileges, which means we could end up regressing on one of ns-process's most lovely features - that it can be run as a non-root user.

Fortunately this can be avoided by making use of setuid. setuid allows a process to run as the user that owns an executable. The idea then is to extract the network setup code into a separate executable, ensure the executable is owned by the root user and to apply the setuid permission on it. We can then call out to the executable from within ns-process (running as a non-root user) as and when we need to. With all this in mind, allow me to introduce netsetgo.

🚦 On your marks, net set, GO!

netsetgo is a small binary that helps to setup Network namespaces for containers. It achieves this by applying the configuration outlined above. For sake of brevity I’m not going to paste the full netsetgo code here, but I will briefly point out the most useful parts so you can take a more detailed look for yourself.

  1. Bridge creation occurs here via a call to netlink.LinkAdd
  2. Veth creation occurs here via another call to netlink.LinkAdd
  3. The veth is attached to the bridge here via a call to netlink.LinkSetMaster
  4. The veth is moved to the new Network namespace here via a call to netlink.LinkSetNsPid
  5. A default route is added to the new Network namespace here via a call to netlink.RouteAdd

In order to make use of netsetgo fromns-process, we’ll need to download the binary and set the correct permissions on it, as follows.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1$ wget "https://github.com/teddyking/netsetgo/releases/download/0.0.1/netsetgo"
$ sudo mv netsetgo /usr/local/bin/
$ sudo chown root:root /usr/local/bin/netsetgo
$ sudo chmod 4755 /usr/local/bin/netsetgo

The 4 in the chmod 4755 signifies that the setuid bit should be set.

👉 Let’s Go

Now that netsetgo is primed and ready it’s time to turn our attention back to ns-process. We need to modify ns-process so that it calls out to netsetgo to configure the network. At first glance this would appear to be relatively simple - we can just create a *exec.Cmd pointing to netsetgo and run it at the appropriate moment?

Of course, nothing’s ever quite as easy as it seems, and here the question of when to run netsetgo requires a bit more thought. Let’s start by looking at how we kick off Namespace creation at the moment (output trimmed for simplicity).

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 4.1
# Filename: ns_process.gofunc main() {
	cmd := reexec.Command("nsInitialisation", rootfsPath)

	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWNS |
			syscall.CLONE_NEWUTS |
			syscall.CLONE_NEWIPC |
			syscall.CLONE_NEWPID |
			syscall.CLONE_NEWNET |
			syscall.CLONE_NEWUSER,
	}

	if err := cmd.Run(); err != nil {
		fmt.Printf("Error running Command - %s\n", err)
		os.Exit(1)
	}
}

Here we’re using cmd.Run() to run a reexec command with a number of CLONE_NEW* flags set. Note that cmd.Run() does not return until the underlying process has exited. Up until now this has been fine because all subsequent namespace configuration has taken place inside the newly-cloned namespaces (via the nsInitialisation func to be specific).

However, netsetgo needs to configure the host’s Network namespace as well as the new one, which means we can no longer rely on the blocking call to cmd.Run().

Fortunately cmd.Run() can be split into two separate calls - cmd.Start() (which returns immediately) and cmd.Wait() (which blocks until the started command exits). This is exactly what we need as it allows us to run netsetgo after the new namespaces have been created but while still executing in the host’s namespaces. Let’s see this in action.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0
# Filename: ns_process.goif err := cmd.Start(); err != nil {
	fmt.Printf("Error starting the reexec.Command - %s\n", err)
	os.Exit(1)
}

pid := fmt.Sprintf("%d", cmd.Process.Pid)
netsetgoCmd := exec.Command(netsetgoPath, "-pid", pid)
if err := netsetgoCmd.Run(); err != nil {
	fmt.Printf("Error running netsetgo - %s\n", err)
	os.Exit(1)
}

if err := cmd.Wait(); err != nil {
	fmt.Printf("Error waiting for reexec.Command - %s\n", err)
	os.Exit(1)
}

Great! This change allows netsetgo to configure the networking across both Network namespaces as required. All that’s left to do now is to ensure that the namespaced /bin/sh process doesn’t start until the network is ready.

Let’s consider the network to be ready once a veth interface has appeared in the new Network namespace. We can use a simple for loop to wait until this is true, as follows.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0
# Filename: net.gofunc waitForNetwork() error {
	maxWait := time.Second * 3
	checkInterval := time.Second
	timeStarted := time.Now()

	for {
		interfaces, err := net.Interfaces()
		if err != nil {
			return err
		}

		// pretty basic check ...
		// > 1 as a lo device will already exist
		if len(interfaces) > 1 {
			return nil
		}

		if time.Since(timeStarted) > maxWait {
			return fmt.Errorf("Timeout after %s waiting for network", maxWait)
		}

		time.Sleep(checkInterval)
	}
}

Here we have a very basic for loop which blocks until either more than one network interface is reported or a timeout of 3 seconds is reached. As the comment mentions, we check for more than one interface as the loopback interface will already exist by default.

Finally, let’s update nsInitialisation to call the above function.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0
# Filename: ns_process.gofunc nsInitialisation() {
	newrootPath := os.Args[1]

	if err := mountProc(newrootPath); err != nil {
		fmt.Printf("Error mounting /proc - %s\n", err)
		os.Exit(1)
	}

	if err := pivotRoot(newrootPath); err != nil {
		fmt.Printf("Error running pivot_root - %s\n", err)
		os.Exit(1)
	}

	if err := waitForNetwork(); err != nil {
		fmt.Printf("Error waiting for network - %s\n", err)
		os.Exit(1)
	}

	nsRun()
}

With all that in place, let’s run the updated Go program.

💁 The following has been tested on Ubuntu 16.04 Xenial with Go 1.7.1

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0$ go build
$ ./ns-process
-[ns-process]- # ifconfig
veth1     Link encap:Ethernet  HWaddr 6A:DD:B4:30:1A:49
          inet addr:10.10.10.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::68dd:b4ff:fe30:1a49/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2359 (2.3 KiB)  TX bytes:578 (578.0 B)-[ns-process]- # route
Kernel IP routing table
Destination     Gateway         Genmask         ... Iface
default         10.10.10.1      0.0.0.0         ... veth1
10.10.10.0      *               255.255.255.0   ... veth1
-[ns-process]- # ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 56 data bytes
64 bytes from 10.10.10.1: seq=0 ttl=64 time=0.098 ms
^C
--- 10.10.10.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.098/0.098/0.098 ms

Much better! We now have a network interface veth1 available and a routable IP address of 10.10.10.2.

☁️ Internet connectivity

Enabling Internet access for ns-process is a little out of scope for this particular article. This is mostly because a lack of Internet connectivity could be the result of any number of things, and attempting to cover all environmental setups would be pretty difficult.

Having said that, the following steps do enable Internet connectivity for ns-process on my generic Ubuntu 16.04 Xenial machine. There’s no guarantee this will work for you, but feel free to try it out if you’re interested.

First up we need to configure a few iptables rules on the host.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0$ sudo iptables -tnat -N netsetgo
$ sudo iptables -tnat -A PREROUTING -m addrtype --dst-type LOCAL -j netsetgo
$ sudo iptables -tnat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j netsetgo
$ sudo iptables -tnat -A POSTROUTING -s 10.10.10.0/24 ! -o brg0 -j MASQUERADE
$ sudo iptables -tnat -A netsetgo -i brg0 -j RETURN

And then we also need to add a DNS nameserver for the namespaced process.

# Git repo: https://github.com/teddyking/ns-process
# Git tag: 5.0$ go build
$ ./ns-process
-[ns-process]- # echo "nameserver 8.8.8.8" >> /etc/resolv.conf
-[ns-process]- # ping google.com
PING google.com (172.217.23.14): 56 data bytes
64 bytes from 172.217.23.14: seq=0 ttl=51 time=4.766 ms

And there we have it - ns-process running with full Internet connectivity.

📺 On the next…

With network configuration complete, ns-process is now setup to configure the User, Mount, Pid and Network namespaces, but what needs to be done about the remaining namespaces? The answer to this and plenty more coming up, stay tuned…

Update: Part 7, “Namespaces in Go - UTS” has been published and is available here.

Golang
Namespaces
Containers
Linux

--

--

Ed King

Written by Ed King

490 Followers

A Software Engineer currently working on and with Kubernetes.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams