MikroTik VRFs — VRF-Lite, Route-Leaking and Mangle Routing-Marks

Few days ago I had the need to make some of my home traffic use a secondary public address when reaching the Internet and I easily managed to do that using MikroTik VRF-lite: a sort of virtual router with it’s own routes and interfaces. In this article I’ll show you how to use a Secondary VRF to make the traffic of some hosts (or only some kinds of traffic) use a secondary public address and we’ll see also how to steer some incoming traffic from the secondary VRF to the main one.

Important Update 06/08/2024: starting from RouterOS 7.14 you can not use physical interfaces as match condition in firewall filter table (with in- and out-interface matches) if those interfaces are placed in a VRF, since the interface that is seen by RouterOS becomes a virtual interface with the same name as the VRF. You can still use them as match condition in mangle and nat tables. So, rules that match on interfaces in vrf must be rewritten in firewall filter table in order to match on other stuff like source/destination IP addresses or on marks applied in the mangle table. More details on the change in the official docs here.

The Big Picture

High level design of the test environment

The test bed for this article is made of the following components:

• A fake public network 10.0.0.0/8 and two private networks 192.168.0.0/24 and 192.168.202.0/24.
• A virtual MikroTik router (CHR — Cloud Host Router) running RouterOS 7.7: it has two wanX interfaces on LAN 10.0.0.0/24 that will be our public interfaces and two ethX interfaces that are connected to the private networks 192.168.0.0/24 and 192.168.202.0/24.
• Two custom nginxdemos/hello:0.3-plain-text containers running on 10.0.0.3 and listening on port 8080/TCP and another one listening on port 8081/TCP: I’ve customised that image in order to show the client IP address that queries the web server and to read CUSTOM_PORT environment variable in order to make nginx listen to port 8080 or 8081. Normally you can change the exposed port with docker directives but I needed to expose the containers on the host network, thus not being able to map the internal port 80 to the port I wanted: I had to use host networking because with the other non-host networks the traffic reaching the container is source-NATted by the gateway of that network and I would miss the client address which is required for my tests.

# cat Dockerfile
FROM nginxdemos/hello:0.3-plain-text
ENV CUSTOM_PORT="8080"
COPY 99_customizations.sh /docker-entrypoint.d/99_customizations.sh
RUN chmod +x /docker-entrypoint.d/99_customizations.sh
COPY listen.conf.template /listen.conf.template
RUN sed -i -e "/.*listen.*::.*/d" /etc/nginx/conf.d/hello-plain-text.conf
RUN sed -i -e "s:.*listen 80.*:include /listen.conf;:g" /etc/nginx/conf.d/hello-plain-text.conf
RUN sed -i -e "s#Server address#Client address: \$remote_addr\nServer address#g" /etc/nginx/conf.d/hello-plain-text.conf

# cat 99_customizations.sh
#!/bin/sh

envsubst < /listen.conf.template > /listen.conf

# cat listen.conf.template
listen $CUSTOM_PORT;

• 3 VMs running Ubuntu: VM201 and VM203 are connected to the 192.168.0.0/24 Network while VM202 is connected to the 192.168.202.0/24 Network and its traffic is Dot1q-tagged with VLAN 202.

Physical Connections

Detail about network connections and VRFs of MikroTik CHR router

This is the physical setup:

• eth0 and eth1 interfaces are put together in a bridge called lan_bridge.
• eth0 is a trunk interface with native VLAN 1 and Dot1q tagged VLAN 202
• eth1 is an access interface on VLAN 1
• VM201 and VM203 are on VLAN 1, the former on eth0 and the latter on eth1
• VM202 is on VLAN 202 on eth0
• lan_bridge interface of the homonymous bridge is the default gateway of network 192.168.0.0/24, has IP address 192.168.0.1 and is within the Main VRF (the default one)
• lan_bridge.202 interface is the default gateway of network 192.168.202.0/24, has IP address 192.168.202.1 and is within the Secondary VRF. This is a vlan interface with vlan-id 202 and parent interface lan_bridge.
• wan0 is the main public interface with IP 10.0.0.200 and is in Main VRF
• wan1 is the secondary public interface with IP 10.0.0.199 and is in Secondary VRF

Traffic Flows

In this section we will see what we want to achieve in terms of traffic flows.

Incoming Traffic

Incoming traffic flows

We want to make the 3 VMs reachable with SSH by exposing ports on the WAN interfaces:

• Traffic directed to 10.0.0.200 port 201 will trigger a destination address/port rewrite via the dstnat chain of the NAT table and it will be DNATted (Destination NATted) to 192.168.0.201 (VM201) on port 22/TCP. The traffic will be fully managed in the Main VRF.
• Traffic directed to 10.0.0.199 port 202 will be DNATted to 192.168.202.202 (VM202) on port 22/TCP. The traffic will be fully managed in the Secondary VRF.
• Traffic directed to 10.0.0.199 port 203 will be DNATted to 192.168.0.203. The traffic arrives on wan1 in Secondary VRF but then it is routed to the 192.168.0.203 (VM203) on port 22/TCP by looking up the route in the Main VRF, thus involving two VRFs to manage the flow of data.

Outgoing Traffic

Outgoing traffic flows

We allow the 3 VMs to reach the Internet as follows:

• VM201 traffic will reach the Internet through wan0 and will be SNATted (Source NATted) on 10.0.0.200.
• VM202 traffic will reach the Internet through wan1 and will be SNATted on 10.0.0.199
• VM203 traffic to port 8080/TCP will reach the Internet through wan1 and will be SNATted on 10.0.0.199. All the other traffic will be SNATted on 10.0.0.200. This will be done via the prerouting chain of the mangle table where we will mark connections to port 8080/TCP in order to move the flow processing in the Secondary VRF.

MikroTik Configuration

In this section we will see the MikroTik configuration used in this lab to implement the traffic flows discussed before.

I’ve inserted comments within the configuration to explain each block.

# Set the hostname of the test mikrotik router to avoid messing up
# with my real router by mistake :) 
/system identity
set name=mik-test-GW

# Rename interfaces
/interface ethernet
set [ find default-name=ether3 ] disable-running-check=no name=eth0
set [ find default-name=ether4 ] disable-running-check=no name=eth1
set [ find default-name=ether1 ] disable-running-check=no name=wan0
set [ find default-name=ether2 ] disable-running-check=no name=wan1

# Create the bridge interface lan_bridge and add eth0 and eth1 to it
/interface bridge
add comment="LAN Bridge" name=lan_bridge vlan-filtering=yes ingress-filtering=yes
/interface bridge port
add bridge=lan_bridge interface=eth0
add bridge=lan_bridge interface=eth1

# Enable tag 202 on eth0 and the lan_bridge (I know, it seems weird but you
# have to enable it also on the lan_bridge interface)
/interface bridge vlan
add bridge=lan_bridge tagged=eth0,lan_bridge vlan-ids=202

# Create vlan interface lan_bridge.202 with vlan ID 202 on lan_bridge interface
/interface vlan
add interface=lan_bridge name=lan_bridge.202 vlan-id=202

# Put the WAN interfaces in the WAN_INTERFACES list
/interface list
add name=WAN_INTERFACES
/interface list member
add interface=wan0 list=WAN_INTERFACES
add interface=wan1 list=WAN_INTERFACES

# Put the LAN interfaces in the LAN_INTERFACES list
/interface list
add name=LAN_INTERFACES
/interface list member
add interface=lan_bridge list=LAN_INTERFACES
add interface=lan_bridge.202 list=LAN_INTERFACES

# Create the SECONDARY_VRF with wan1 and lan_bridge.202 interfaces
/ip vrf
add interfaces=wan1,lan_bridge.202 name=SECONDARY_VRF

# Set the IP addresses on the interfaces
/ip address
add address=10.0.0.200/24 interface=wan0 network=10.0.0.0
add address=10.0.0.199/24 interface=wan1 network=10.0.0.0
add address=192.168.0.1/24 interface=lan_bridge network=192.168.0.0
add address=192.168.202.1/24 interface=lan_bridge.202 network=192.168.202.0


# ----------------------------------------------------------------------
# ROUTING TABLE
# ----------------------------------------------------------------------
/ip route
# This is the default route in the Main VRF
add gateway=10.0.0.254
# This is the default route in Secondary VRF: we set the route in the routing-table 
# SECONDARY_VRF with a next-hop (gateway) that we say to be reachable via the
# Secondary VRF by specifying @SECONDARY_VRF. This makes the router use wan1
# to reach to the 10.0.0.0/24 network on which the next-hop is found
add gateway=10.0.0.254@SECONDARY_VRF routing-table=SECONDARY_VRF
# Since VM203 is connected to the Main VRF via eth1, when incoming traffic
# to 10.0.0.199 in Secondary VRF is DNATted to 192.168.0.203 destination IP
# we must tell the router how to reach that address because it is reachable
# only in the Main VRF. This is why we append @main to the gateway, which is
# set equal to the destination address itself. If 192.168.0.0/24 would be reachable
# in Main VRF through another network, let's say 172.16.0.0/24 with next-hop
# 172.16.0.2 we would have set gateway=172.16.0.2@main
add dst-address=192.168.0.203 gateway=192.168.0.203@main routing-table=SECONDARY_VRF
# Since 192.168.0.203 can be reached directly through lan_bridge in main routing
# table we could replace the previous routing entry with the following one
# that forces the router to try to reach directly 192.168.0.203 via an ARP
# request on lan_bridge interface
# add dst-address=192.168.0.203 gateway=lan_bridge routing-table=SECONDARY_VRF


# FIREWALL RULES
# ----------------------------------------------------------------------
# NAT TABLE
# ----------------------------------------------------------------------
/ip firewall nat
# Source NAT outgoing traffic
add action=masquerade chain=srcnat out-interface-list=WAN_INTERFACES

# Implement Destination NAT for incoming traffic
# Traffic to the primary public IP on wan0
add action=dst-nat chain=dstnat dst-port=201 in-interface=wan0 protocol=tcp to-addresses=\
    192.168.0.201 to-ports=22
# Traffic to the secondary public IP on wan1
add action=dst-nat chain=dstnat dst-port=203 in-interface=wan1 protocol=tcp to-addresses=\
    192.168.0.203 to-ports=22
add action=dst-nat chain=dstnat dst-port=202 in-interface=wan1 protocol=tcp to-addresses=\
    192.168.202.202 to-ports=22

# ----------------------------------------------------------------------
# FILTER TABLE
# ----------------------------------------------------------------------
/ip firewall filter
# We accept all the already established/related connections that traverse the
# router. We have two rules, the first one to use fasttrack if available and
# the second one as a fall-back
add action=fasttrack-connection chain=forward comment="FastTrack Established/Related" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Allow Established/Related" connection-state=\
    established,related

# Accept input packets belonging to established/related connections
add action=accept chain=input comment="Allow Established/Related" connection-state=established,related

# Allow incoming management traffic: avoid that on the public interfaces of
# a real router, this is just to easily manage this test router. On a real router
# you would add in-interface-list=LAN_INTERFACES, for example.
add action=accept chain=input dst-port=22,80 protocol=tcp

# Default drop for all the other input traffic on the WAN_INTERFACES
add action=drop chain=input in-interface-list=WAN_INTERFACES

# Allow all the traffic forwarded from LAN to WAN
add action=accept chain=forward in-interface-list=LAN_INTERFACES

# Allow traffic forwarded from Internet to the VMs. The destinations in the
# following rules are post-DNAT since DNAT already happened:
# PRE-ROUTING (DNAT) -> ROUTING DECISION -> FILTERING -> POST-ROUTING (SNAT)
add action=accept chain=forward comment=\
    "Allow traffic to 10.0.0.200:201 (After DNAT to 192.168.0.201:22)" dst-address=\
    192.168.0.201 dst-port=22 in-interface=wan0 protocol=tcp
add action=accept chain=forward comment=\
    "Allow traffic to 10.0.0.199:203 (After DNAT to 192.168.0.203:22)" dst-address=\
    192.168.0.203 dst-port=22 in-interface=wan1 protocol=tcp
add action=accept chain=forward comment=\
    "Allow traffic to 10.0.0.199:202 (After DNAT to 192.168.202.202:22)" dst-address=\
    192.168.202.202 dst-port=22 in-interface=wan1 protocol=tcp

# Log and drop the other traffic from WAN to LAN
add action=log chain=forward in-interface-list=WAN_INTERFACES log-prefix=FWD_DROP
add action=drop chain=forward in-interface-list=WAN_INTERFACES

# ----------------------------------------------------------------------
# MANGLE TABLE
# ----------------------------------------------------------------------
/ip firewall mangle
# Mark the connections from VM203 to the internet with destination port 8080/TCP
add action=mark-connection chain=prerouting connection-state=new dst-port=8080 in-interface=\
    lan_bridge new-connection-mark=TO-SECONDARY-VRF protocol=tcp src-address=192.168.0.203

# Force the marked traffic to be managed (i.e. have a destination lookup) in the
# Secondary VRF
add action=mark-routing chain=prerouting connection-mark=TO-SECONDARY-VRF new-routing-mark=\
    SECONDARY_VRF

Testing The Implementation

Incoming Traffic

We connect to the 3 VMs via SSH and check that everything works as expected: we simply start a non-interactive SSH session (-T) and check the hostname and IP address of the target host by passing shell commands within the double-quotes.

# Connect to VM201 on 10.0.0.200 port 201
ssh 10.0.0.200 -l ubuntu -p 201 -T "hostname ; ip address show dev eth0 | egrep 'eth0|inet '"                         ─╯
ubuntu-vm1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.0.201/24 brd 192.168.0.255 scope global eth0

# Connect to VM202 on 10.0.0.199 port 202
ssh 10.0.0.199 -l ubuntu -p 202 -T "hostname ; ip address show dev eth0 | egrep 'eth0|inet '"                         ─╯
ubuntu-vm2
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.202.202/24 brd 192.168.202.255 scope global eth0

# Connect to VM2023 on 10.0.0.199 port 203
ssh 10.0.0.199 -l ubuntu -p 203 -T "hostname ; ip address show dev eth0 | egrep 'eth0|inet '"                         ─╯
ubuntu-vm3
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.0.203/24 brd 192.168.0.255 scope global eth0

Outgoing Traffic

We use again the SSH connection to launch an HTTP request from each VM to the hello nginx container and we look at the IP address seen from the server. It must be 10.0.0.200 for the connection from VM201 and 10.0.0.199 for the connections from VM202. For the connections from VM203 it must be 10.0.0.199 when destination port is 8080/tcp and 10.0.0.200 for all the other traffic.

# HTTP request from VM201: client address seen by the server is 10.0.0.200
ssh 10.0.0.200 -l ubuntu -p 201 -T "hostname ; curl http://10.0.0.3:8080 --silent | grep address"                     ─╯
ubuntu-vm1
Client address: 10.0.0.200
Server address: 10.0.0.3:8080

# HTTP request from VM202: client address seen by the server is 10.0.0.199
ssh 10.0.0.199 -l ubuntu -p 202 -T "hostname ; curl http://10.0.0.3:8080 --silent | grep address"                     ─╯
ubuntu-vm2
Client address: 10.0.0.199
Server address: 10.0.0.3:8080

# HTTP request from VM203 to port 8080/TCP: client address seen by the server 
# is 10.0.0.199
ssh 10.0.0.199 -l ubuntu -p 203 -T "hostname ; curl http://10.0.0.3:8080 --silent | grep address"                     ─╯
ubuntu-vm3
Client address: 10.0.0.199
Server address: 10.0.0.3:8080

# HTTP request from VM203 to port 8081/TCP: client address seen by the server 
# is 10.0.0.200
ssh 10.0.0.199 -l ubuntu -p 203 -T "hostname ; curl http://10.0.0.3:8081 --silent | grep address"                     ─╯
ubuntu-vm3
Client address: 10.0.0.200
Server address: 10.0.0.3:8081

Everything works as expected :)

Lab Setup Info

Everything has been done in a virtual environment running on Proxmox VE 7.3:

• Mikrotik wan0 and wan1 interfaces are both connected to vmbr0, the main bridge of Proxmox VE
• Mikrotik eth0 is connected to vmbr200
• Mikrotik eth1 is connected to vmbr201.
• Vmbr200 is vlan-aware and has VM201 connected to it on native VLAN 1 and VM202 on Dot1q Tagged VLAN 202.
• vmbr201 has VM203 connected to it on native VLAN1.
• Nginxdemos/hello custom docker containers are running on host network on server 10.0.0.3 on the 10.0.0.0/24 network that is reached by Proxmox VE host via its ethernet interface eno1 in vmbr0.

Reference

This is the reference page about VRFs on the Official MikroTik Documentation: Virtual Routing and Forwarding (VRF)

Conclusions

I hope you’ve found something useful in this article, I find MikroTik routers extremely powerful and full of features, given the very low price they have. Furthermore, you can freely download virtual router images to test configurations in a virtual environment with the only limit of 1Mbps throughput, more than sufficient for this sort of testing (you can also license them for higher throughput if needed). This lab has been setup in very few time thanks to the amazing OpenSource software ProxmoxVE running in my HomeLab: more details about my HomeLab in my previous article Intel NUC 10/Proxmox VE HomeLab.