Guide: Firewall and router with Proxmox

Firewall and router with Proxmox

By default Proxmox does not come with a firewall, which may leave it and your virtual servers exposed to the elements of the Internet.

An additional issue arises when a hosting provider blocks servers if unauthorized MAC addresses are detected. As Proxmox’s bridged network creates and exposes MAC addresses for its virtual network interfaces, this may cause your server to be blocked from the hosting provider’s network.

To combat both this article will describe how to create your own virtual network with firewall protection using Shorewall, a popular and effective firewall / router software package.

Overview

We will be creating three separate zones, namely:

  1. The Internet (“net”);
  2. The firewall / Proxmox Host Node (“fw”); and
  3. The virtual network (“dmz”)

All traffic from the Internet is filtered or blocked by the firewall, after which it is routed to its final destination on the virtual network or the Proxmox Host Node itself. A visual representation of this can be seen in the figure 1.


Host Network Configuration

Some modifications are required to your host network configuration, particularly the default vmbr0 network interface. We will turn it into a blind bridge (without any bride ports) and assign it an IP address within the private IP range of 10.0.0.0/8 (A-Class).

Determine the current network configuration

Before we do this, we need to determine the current network configuration because this can be different depending on the hosting provider and other factors. Assuming that eth0 is the network interface that connects the server to the Internet, we issue the following command:

ifconfig eth0

This will give an output similar to:

eth0      Link encap:Ethernet  HWaddr 00:ff:ff:ff:ff:ff
inet addr:91.11.22.33 Bcast:91.11.22.255 Mask:255.255.255.0
...

It gives us the current IP address, broadcast address and netmask used by eth0. One last piece of information we need is the gateway used by eth0, which is obtained with the follwing command:

route -n

You will see an output similar to:

...
0.0.0.0 91.11.22.254 0.0.0.0 UG 0 0 0 eth0

The first column of 0.0.0.0 designates the default route (any traffic that has no specific route), and the second column the gateway. Now that we have obtained all this information, we can edit the /etc/network/interfaces file.

First we need to verify that eth0 has already been defined within this file. It will look similar to:

auto eth0
iface eth0 inet static
address 91.11.22.33
netmask 255.255.255.0
broadcast 91.11.22.255
gateway 91.11.22.254
...(additional stanzas)...

Where the IPs match those you have obtained by use of the ifconfig and route commands.

Or if your server uses DHCP to assign the IP address then it will look similar to:

allow-hotplug eth0
iface eth0 inet dhcp
…(additional stanzas)…

If either is the case, you can skip the follow section and continue to change the vmbr0 network interface.

Updating the eth0 network interface

If you have reached this section, then your vmbr0 network interface was most likely directly bridged with your eth0 network interface, meaning that vmbr0 contains your public IP address, network gateway and other settings. Because we will turn vmbr0 into a blind bridge in the next section, we need to create or edit a separate eth0 stanza in the /etc/network/interfaces file first.

In the previous steps we have obtained all the information required for this purpose (using the ifconfig and route commands). This information may already be present in your vmbr0 stanza as well, in which case you can use this instead.

We edit the eth0 stanza to look as following:

auto eth0 iface
eth0 inet static
address 91.11.22.33
netmask 255.255.255.0
broadcast 91.11.22.255
gateway 91.11.22.254

Where the example IPs are replaced by the actual address, netmask, broadcast and gateway IPs found in the previous steps or obtained from the current vmbr0 stanza.

WARNING! It is important to understand that editing the eth0 stanza can lead to an inability to connect to the server if done incorrectly. If you rely on remote access such as SSH, or if you are uncertain about the information that needs to be entered in the various fields, please contact your administrator or hosting provider for assistance. As always, use care and make backups of existing files.

Change the vmbr0 network interface

The vmbr0 stanza in the /etc/network/interfaces will be changed to look like:
… (original configuration) …

auto vmbr0
iface vmbr0 inet static
address 10.254.254.254
netmask 255.0.0.0
broadcast 10.255.255.255
bridge_ports none
bridge_stp off
bridge_fd 0

The significant changes here are the IP addresses used for the address, netmask and broadcast as well as changing bridge_ports to “none”. The IP address of 10.254.254.254 can be any in the 10.0.0.0/8 range and so may be changed if you wish. It will later be used as the gateway address for the virtual servers.

The changes can be applied without having to reboot the system using:

/etc/init.d/networking restart

Shorewall

Installing Shorewall is simply a matter of executing the following command:

apt-get install shorewall

If you wish to use shorewall with IPv6 capabilities, a few additional steps will be required. At the moment of writing, Debian has not included the latest version of Shorewall6 into its main package source. You will therefore need to add the SID (unstable branch) to your available package sources:

echo “deb http://ftp.fr.debian.org/debian sid main” >> /etc/apt/sources.list
aptitude update

Once this has been done, Shorewall with IPv6 support — aptly named shorewall6 — can be installed with:

apt-get install shorewall6

NOTE! It is important to note that Shorewall6 makes a strict distinction between IPv4 and IPv6 and each has to be configured individually. That is, the files located in /etc/shorewall/ are for IPv4 only and /etc/shorewall6/ contains files related to IPv6 only. The IPv4 version cannot control what will happen on IPv6 and vice versa!

Most rules and policies in for Shorewall are directly transferable between /etc/shorewall/ and /etc/shorewall6/. Due to this, this article will only focus on IPv4 configuration with one exception:

You will want to edit the /etc/shorewall/shorewall.conf file, and change the following value from:

DISABLE_IPV6=Yes

to:

DISABLE_IPV6=No

If this is not done, any configuration done in /etc/shorewall6/ would be ignored (and will also disable any existing IPv6 traffic).

Enable IP Forwarding

First, a minor modification to the Shorewall application configuration is made to enable IP forwarding. This will permit some functions that will be discussed later.

Edit /etc/shorewall/shorewall.conf and change the following value from:

IP_FORWARDING=Off

to:

IP_FORWARDING=On

Zones

The /etc/shorewall/zones file is created to establish the zones names and what type of zones they are:

#ZONE   TYPE            OPTIONS         IN                      OUT
# OPTIONS OPTIONS
fw firewall
net ipv4
dmz ipv4

The fw zone is a self-reference to the server on which Shorewall is running. Note that this will not include the Proxmox virtual servers; you have to consider this as an entirely separate server in this sense.

Interfaces

The network interfaces on the server need to be defined and assigned to a specific zone. The net zone will be assigned to the eth0 network interface and will designate all traffic coming from or going to the internet. The dmz zone will be the internal zone for the virtual network, which will contain the virtual servers that either use venet or veth network interfaces.

Create the /etc/shorewall/interfaces file and add the following:

#ZONE   INTERFACE       BROADCAST       OPTIONS
net eth0 detect blacklist,nosmurfs
dmz venet0 detect routeback
dmz vmbr0 detect routeback,bridge

Policy

A base policy needs to defined for each one of the zones. It specifies the default actions on in- and outgoing traffic, and in this article the following policies will be defined:

Traffic from the firewall to:

  • the internet is permitted
  • DMZs is permitted
  • other processes on the firewall is permitted

Traffic from the DMZ (virtual servers) to:

  • another virtual server is permitted
  • the internet is permitted
  • the firewall is denied and 1 information message per second (with a burst of 2) will be record when access is attempted.

Traffic from the internet to:

  • the firewall is denied
  • DMZs is denied, generating 8 messages per second (with a burst of 30 messages) whenever access is attempted.

Any traffic not defined in any of the zones (either by accident or purposely) will be rejected.

To do this, we will create the /etc/shorewall/policy file:

#SOURCE DEST    POLICY          LOG     LIMIT:          CONNLIMIT:
# LEVEL BURST MASK
# From Firewall Policy
fw fw ACCEPT
fw net ACCEPT
fw dmz ACCEPT
# From DMZ Policy
dmz     dmz     ACCEPT
dmz net ACCEPT
dmz fw DROP info
# From Net Policy
net fw DROP info
net dmz DROP info
# THE FOLLOWING POLICY MUST BE LAST
#
all all REJECT info

For those who have followed this guide before and are experiencing some performance issues, please remove the limit burst options after “info”, ie “1/sec:2”. The reason is that this takes priority over rules; this was a painful discovery on my end!

Basic Rules

The policy defined earlier will deny any traffic coming from the internet to the firewall, which will include the SSH service and the Proxmox web-based manager. Since this is undesirable, a few rules need to be created that override this base policy.

Create the /etc/shorewall/rules file:

#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE
# Permit access to SSH
SSH/ACCEPT net fw — — — — 6/min:5
# Permit access to Proxmox Manager and Console
ACCEPT net fw tcp 443,5900:5999
# PING Rules
Ping/ACCEPT all all
# LAST LINE — DO NOT REMOVE

As you may notice, there is also an additional rule for ping. For testing purposes, it would be wise to permit a ping from and to any of your zone, including the internet.

The SSH/ACCEPT rule is in fact a macro that comes with Shorewall. You could also define the same rule as following:

ACCEPT net fw tcp 22 — — 6/min:5

Also, at the very end of the SSH rule you notice “6/min:5”. This specifies the connection rate and in this case it reduces the connection rate to 6 per minute (1 per 10 seconds) with a maximum initial burst of 5. It is added here to slow down brute force SSH attacks.

Testing Configuration

After creating the files, your /etc/shorewall/ directory might look similar to:

drwxr-xr-x  2 root root 4096 2009-07-01 06:36 .
drwxr-xr-x 82 root root 4096 2009-07-06 10:03 ..
-rw-r--r-- 1 root root 522 2009-06-26 20:05 interfaces
-rw-r--r-- 1 root root 453 2007-11-15 23:24 Makefile
-rw-r--r-- 1 root root 781 2009-06-26 21:16 policy
-rw-r--r-- 1 root root 2355 2009-07-02 22:42 rules
-rw-r--r-- 1 root root 4134 2009-06-20 21:58 shorewall.conf
-rw-r--r-- 1 root root 438 2009-06-26 20:04 zones

Before using the configuration you will want to test it first, particularly to make sure you are not blocking SSH access. Issue the following command:
shorewall try /etc/shorewall 60

The parameter 60 refers to 60 seconds. Shorewall will use the configuration located in /etc/shorewall/ for 60 seconds and then reverts to the previous settings (or no firewall).

After issuing the command, establish a new connection to your server using SSH and check whether your Proxmox web-based manager is accessible. If you are receiving error messages from Shorewall or you are unable to access SSH during the 60-second test period, please verify the configuration and try again.

Starting Shorewall

By default Shorewall is not enabled during boot time as a safety precaution for first-time configurations. To enable it, edit /etc/default/shorewall file and change:

startup=0

to:

startup=1

You can start Shorewall manually with:

shorewall start

And after making any changes to the Shorewall configuration, issue the command:

shorewall restart

It is best not to use /etc/init.d/shorewall restart. Doing so will temporarily disable the firewall, which permits access to normally blocked ports.

Although this period is brief, it may still be enough time for someone to establish a connection to the server — Shorewall does not block existing connections by default.


Proxmox

Once Shorewall has been configured, there will be three distinct zones on the Proxmox server:

  • the Firewall / Proxmox host at fw
  • the virtual network zone for virtual servers at dmz
  • the internet at net

IP Assignment

To further separate the internet and virtual servers as distinct areas, each virtual server will be assigned an IP address in the 10.0.0.0/8 range (10.0.0.1–10.255.255.254).

The exception is that one can no longer use 10.254.254.254 as this has been assigned to the vmbr0 network interface earlier in this guide.

Outgoing internet traffic

Due to this separation and the use of A-class (10.0.0.0/8) IP addresses, outgoing traffic from a virtual server to the internet needs to be translated (so that Shorewall and other Internet routers know where to send responses to).

This will be defined in the /etc/shorewall/masq file.

In its simplest form, /etc/shorewall/masq can be set to the follwing:

#INTERFACE      SOURCE          ADDRESS         PROTO   PORT(S) IPSEC   MARK
eth0 10.0.0.0/8
# LAST LINE -- DO NOT REMOVE

This means that all traffic originating from 10.0.0.0/8 and going to the internet will pass through the eth0 network interface using the IP address assigned to eth0.

If you wish to make all traffic appear from a particular IP addresses, it can be specified as the third parameter. For example:

eth0 10.0.0.0/8 91.121.0.1

Or perhaps there’s a specific internal IP address that must appear externally as another IP address, you can do this as folowing:

+eth0 10.0.1.101 91.121.0.2
eth0 10.0.0.0/8 91.121.0.1

Notice the plus (‘+’) sign in front of eth0. All traffic from 10.0.0.0/8 will appear to be coming from IP 91.121.0.1, except traffic coming from 10.0.1.101 will appear as coming from 91.121.0.2.

Incoming internet traffic

The separation between the internet and virtual servers not only applies to outgoing traffic, but also incoming traffic. There are two methods of directing incoming traffic,which is Proxy ARP or DNAT. This article will focus on DNAT; for more information on Proxy ARP and Shorewall, visit http://www.shorewall.net/manpages/shorewall-proxyarp.html.

For example, to forward HTTP traffic on any external IP address to a virtual server with the assigned IP of 10.0.1.101, edit the /etc/shorewall/rules file as following:

...(existing rules)...
DNAT net dmz:10.0.1.101 tcp 80

The added benefit of DNAT is that a single IP address can be used for multiple virtual servers, provided that the traffic is on a different port. For example, HTTP traffic on external IP address 91.121.0.1 may be sent to a virtual server with the assigned IP of 10.0.1.101, whereas FTP traffic may be sent to a virtual server with assigned IP of 10.0.1.102 instead:

…(existing rules)…
DNAT net dmz:10.0.1.101 tcp 80–91.121.0.1
DNAT net dmz:10.0.1.102 tcp 21,23–91.121.0.1

It is even possible to route traffic to a different internal port. For example, to forward HTTP traffic on external IP address 91.121.0.1 to a virtual server with the assigned IP of 10.0.1.103 and listening on 8180:

…(existing rules)…
DNAT net dmz:10.0.1.103:8180 tcp 80–91.121.0.1

Bridged Networking

The venet network interface is certainly the simplest method to use in Proxmox. However, venet is not available in KVM (fully virtualized servers) and there may be another reason why you might want to use the veth network interfaces with regular containers (such as the use of DHCP).

For this reason the vmbr0 network interface on the host was reconfigured to use the IP address of 10.254.254.254. It will act as the gateway entry for those virtual servers using veth network interfaces.

Although additional configuration needs to be done within a virtual server, you can use the same Shorewall rules for in- and outgoing traffic as described earlier (ie., DNAT or outgoing traffic).

Linux (Debian)

Inside a Debian Linux virtual server, you will specify /etc/network/interfaces as following:

auto eth0
iface eth0 inet static
address 10.0.1.101
netmask 255.0.0.0
gateway 10.254.254.254

where 10.0.1.101 is the IP address to be used by this particular virtual server.

If you are using both venet and veth network interfaces at the same time, as may be the case with certain IPv6 configurations, the file /etc/network/interfaces.tail should be used instead.

Microsoft Windows

For networking within Windows, proceed to your Networking control panel (or the Network and Sharing Center). Select the appropriate Local Area Connection and right-click to reveal the Properties menu option. UAC (User Account Control) may request your permission to proceed.

In the list of This connection uses the following items, select Internet Protocol (TCP/IP) (or Internet Protocol Version 4 (TCP/IPv4)). Click the Properties button.

At the General tab, change the following selections:
Use the following IP address:

IP address: 10.0.1.101
Subnet mask: 255.0.0.0
Default gateway: 10.254.254.254

Use the following DNS server addresses:

Preferred DNS server: xxx.xxx.xxx.xxx
Alternate DNS server: yyy.yyy.yyy.yyy

Where xxx… and yyy… are your preferred DNS servers.

Non-private IP Assignment

The setup as described above has separated your virtual servers from the internet by use of a zone (dmz) and A-class IP range (10.0.0.0/8). However, it is still possible to assign a non-private IP directly to one of your virtual servers if the venet network interface is used.

Internet traffic (from the net to the dmz zone) will still be blocked per the policy established in the above setup, and you will need to add additional rules to your Shorewall configuration. The major difference is that you must use ACCEPT instead of DNAT.

For example, let’s assume the IP address 91.121.0.1 was directly assigned to a virtual server. To permit internet Web traffic (port 80) to this container, add the following rule to your /etc/shorewall/rules file:

ACCEPT net dmz:91.121.0.1 tcp 80

Please note that for veth network interfaces (bridged) Proxy ARP is required.


Extra Hardening

Following are considered advanced topics to further enhance the firewall and Proxmox. They may not be required in all situations.

Restricting IP Addresses per MAC

When the veth bridged networking is used in Proxmox, the virtual server will have a fully emulated network interface with its own MAC address. The downside is that Proxmox cannot assign the IP address directly and so you will have to configure this within the virtual server.

This should not be an issue if you have full control over the virtual server. However, if the virtual server belongs to a customer or in a worst case scenario is compromised, you no longer have full control over it and it will be possible to change or assign additional IP addresses to the virtual server.

Although earlier in this article containers have already been separated from the internet, it would still be possible to use a private IP address or network interface of an already existing other container. It may therefore be possible to spoof another virtual server.

To implement additional safeguards against this, Shorewall needs a few extra rules described here.

Enable the maclist

First the maclist option needs to be enabled for the interface we will monitor MAC / IP relations on. Your /etc/shorewall/interfaces may currently look like this:

#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect blacklist,nosmurfs
dmz venet0 detect routeback
dmz vmbr0 detect routeback,bridge
# LAST LINE — DO NOT REMOVE

We will add the maclist option to vmbr0 in the dmz zone, ending up with:


dmz vmbr0 detect routeback,bridge,maclist
# LAST LINE — DO NOT REMOVE

Defining MAC / IP relation

Now we need to obtain the MAC address of the veth interface used in the virtual server and there are several methods for this. We can obtain the MAC address from within the container by issuing the command:

ifconfig | grep eth

which produces an output similar to:

eth0 Link encap:Ethernet HWaddr 00:18:51:f9:43:1e

Alternatively, we can issue the following command from the host node (Proxmox server):

cat /etc/vz/conf/<VEID>.conf | perl -lne ‘print for /mac=(.*),host_ifn/g’

Or in case it is a fully virtualized server (KVM):

cat /etc/qemu-server/<VEID>.conf | perl -lne ‘print for /virtio=(.*)$/g’

where <VEID> is the number of the virtual server (container) for which you wish to obtain the MAC address. This will produce an output similar to:

00:18:51:F9:43:1E

Once we have obtained the MAC address, we create (or edit) the /etc/shorewall/maclist file as following:

# DISPOSITION INTERFACE MAC IP
ACCEPT vmbr0 00:18:51:f9:43:1e 10.0.1.101
# LAST LINE — DO NOT REMOVE

In this example, the MAC address has been provided in our earlier examples and the IP address 10.0.1.101 is the only permitted IP address that can be used by this virtual server.

Apply the changes by issuing the command

shorewall restart

At this point, if the virtual server changed the IP address (or added another IP address) other than 10.0.1.101, that traffic will be rejected as per the default behaviour of a Shorewall installation on Proxmox/Debian. If this behaviour needs to be changed, it can be set in the /etc/shorewall/shorewall.conf file with this variable:

MACLIST_DISPOSITION=REJECT

Additonal Notes

This guide is only intended to provide basic protection for your virtual servers and the Proxmox host node. No guarantees or warranties are implied, and you should always remain vigilant against potential network intrusions (in other words: do not rely on a firewall alone).

You may also wish to configure Shorewall according to your particular needs. For example, in this guide virtual servers are permitted to connect to each other within the dmz zone and could pose a risk. You may wish to shield the virtual servers from each other in case one of them has been compromised (hint: edit the policy).

The Shorewall website includes numerous examples in its documentation that may help you further.