Creating Instances with Multiple Network Interfaces and IP route issue on Google Cloud Platform

If you want to assign two Ethernet interfaces to the Google Cloud Compute engine instance, you must create the both interfaces while you are creating the instance. You can’t add it later.

You can read https://cloud.google.com/vpc/docs/create-use-multiple-interfaces . But you need to configure something else to make it work permanently.

Here is my notes to configure multiple interfaces. I will be using Debian 9 in this example.

Create a new VPC

You need to create a ne VPC for your region and then create a different subnet from the default one. I assume that you created new VPC second and then subnet 10.150.0.0/20

To make it easy, I enabled auto subnet creation for the second VPC then deleted a subnet 10.150.0.0/20in another region that I will never use and deleted subnet in my region that has the same network for default VPC. Finally assigned 10.150.0.0/20 to my new VPC region
Create a new instance and create two interfaces under Networking menu. It is better to assign External Static IP

Create a Firewall rule

GCP will not create a firewall rule for your VPC that will be used for second ethernet. So you must go to the VPC Network ->Firewall Rules then create a new rule for the ports you want to allow.

Create Route policy

When you boot the Linux instance, you will see two ethernet interfaces in ip addr output. but you will not able to access via External IP of the second interface. Even if you configure to GCP firewall. You need to add ip route and rule to reply packets from eth1 instead of eth0.

I assume that your second ethernet interface is 10.150.0.2

sudo ifconfig eth1 10.150.0.2 netmask 255.255.255.255 broadcast 10.150.0.2 mtu 1430
sudo echo "1 rt1" | sudo tee -a /etc/iproute2/rt_tables
sudo ip route add 10.150.0.1 src 10.150.0.2 dev eth1 table rt1
sudo ip route add default via 10.150.0.1 dev eth1 table rt1
sudo ip rule add from 10.150.0.2/32 table rt1
sudo ip rule add to 10.150.0.2/32 table rt1

This routing will work fine until you reboot the instance.

Configure Route/Rule to run at startup

Now the nightmare start!

If I were using a Linux box on my laptop or on Virtualbox, I just needed to add /etc/network/interface like the following.

auto eth1
iface eth1 inet dhcp
up ip route add 10.150.0.1 src 10.150.0.2 dev eth1 table rt1
up ip route add default via 10.150.0.1 dev eth1 table rt1
up ip rule add from 10.150.0.2/32 table rt1
up ip rule add to 10.150.0.2/32 table rt1

But wait! There is no eth1 configuration in the interfaces file on GCP It contains config just for eth0!

# cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto eth0
iface eth0 inet dhcp
  • Even if you add eth1 config above into interfaces file. It will not work
  • I decided to add /etc/rc.local after enabling rc-local.service. No luck!
  • Configured startup script to run ip route/ip rule command https://cloud.google.com/compute/docs/startupscript it did not work!
    I got startup-script: INFO startup-script: RTNETLINK answers: Network is down error!

Solution

it is there https://github.com/GoogleCloudPlatform/compute-image-packages/issues/475

Root cause: GCP uses /usr/bin/google_network_daemon service to configure second interface and it does not take care of startupscript, /etc/rc.local and interfaces file.

Edit /etc/default/instance_configs.cfg and disable NetworkInterfaces setup by setting setup = false

# vi /etc/default/instance_configs.cfg
[NetworkInterfaces]
dhclient_script = /sbin/google-dhclient-script
dhcp_command =
ip_forwarding = true
setup = false

then add the following files into /etc/network/interfaces

auto eth1
iface eth1 inet dhcp
up ip route add 10.150.0.1 src 10.150.0.2 dev eth1 table rt1
up ip route add default via 10.150.0.1 dev eth1 table rt1
up ip rule add from 10.150.0.2/32 table rt1
up ip rule add to 10.150.0.2/32 table rt1

Ismail YENIGUL

Devops Engineer at Feedstock Inc.