Lopsided routing, a stealthy hole punch into FortiGate
A critical infrastructure player found a way for traffic to leak out from their isolated network and asked us for help. They had updated their SensorFu Beacon to a version that included a new IP payload test. After the update Beacon alerted the customer that IP protocol number 132, also known as SCTP protocol, is leaking out of the isolation to the Internet. Our products are good in finding these types of leaks, and typically they can be fixed in 15 minutes. But this time there were no quick fixes. This customer had a FortiGate firewall with default blocking policy enabled, really strict firewall rules in general, and there were no exceptions related to SCTP traffic.
Since configuration looked fine on the surface, we started to dig deeper in order to find the elusive root cause in a joint investigation with the Customer’s service provider.
We set up a simple test network with two virtual Linux servers running Beacon and Beacon Home applications, and a virtual FortiGate firewall in between.
First we wanted to validate the correct functionality of the FortiGate firewall in its default configuration. The firewall has a default deny all policy which should not let anything pass.
After leaving the Beacon running overnight we could confirm that there were no leaks.
We reported our results to our customer but the question still remained. Why are we not able to reproduce the leak?
Enter Asymmetric Routing
We got a hint from the service provider that the firewall is running in asymmetric routing mode. In asymmetric routing, the outbound packets and return packets of traffic flow travel different routes. This can happen, for example, in a redundant firewall setup. Asymmetric routing can cause problems when combined with stateful firewalls. If the firewall only sees the return packets, they are dropped as the firewall has no prior state information about the flow.
The FortiGate firewall can be configured to permit the return packets of asymmetrically routed flows to pass. We enabled this option while keeping everything else the same.
After a couple of hours we checked the results and found out that IP protocol 132, SCTP, was now indeed leaking through the firewall.
The first assumption was that the asymroute option changes the behaviour of the default deny policy, so we added explicit rules to deny all incoming traffic on the port the Beacon was connected to, and another rule to deny IP protocol 132. Even so, the firewall was still leaking the SCTP traffic.
Then we found a technical note describing the asymmetric routing option in more detail, especially the handling of TCP and ICMP packets:
If the packet is not a SYN and the session doesn’t exist (asymmetric routing), then all packets are passed to the CPU and the FortiGate doesn’t lookup for matching firewall policies.
For ICMP packets the rule is the same as for TCP packets. When asymmetric routing is enabled, if the ICMP packet is not a request and the session doesn’t exist on the FortiGate, the ICMP reply will be routed if a route exists on the routing table without security inspection.
Turns out that enabling asymmetric routing allows the return packets of TCP and ICMP flows to bypass the firewall rules completely. What the technical note does not mention is that all SCTP packets also seem to bypass the firewall.
Beacon’s IP payload escape works by changing the protocol number in the IP header and sending it with the escape payload. Since the packets leak even though they are not actually valid SCTP packets, it is possible to bypass the firewall just by spoofing the protocol number.
This means that in this setup with the asymmetric routing option enabled, there is currently no way to completely isolate networks connected via this firewall product. Your mileage may vary if your setup has been configured for high availability or router redundancy and firewall states are properly synchronized between the gateway devices. Unfortunately with generic asymmetric routing you are in for a surprise, and it is used in the wild, so please check your setup.
PS. After turning asymmetric routing off again, SCTP leak stopped, but the firewall rule denying IP protocol 132 did not get any hits and the logs did not tell anything about the blocked SCTP packets. I bet this sort of stealth mode should not be there by design. :)
