OpnSense Firewall Configuration: Performance Tuning for Multi-Gigabit Internet and better speeds in general

4 min readMar 30, 2024

Who doesn’t love faster and better performing internet? Especially when we are running high performance hardware?

Lets go on a journey of tuning our OpnSense Firewall for better performance.

Hardware offload good or bad? Trash?

It seems like the general consensus is, somewhat counter intuitively, that you should not enable Hardware TSO or Hardware LRO on a firewall appliance.

Trying each one of these interface settings individually, occasionally I would see some performance gains (Hardware LRO gave me a noticeable performance boost), but some of the settings also damaged performance.

I also noticed some strange issues with the hardware offloading, such as the hardware offloading would help the LAN side perform significantly better, but the performance on the WAN side would take a nosedive.

OPNsense Tunables (sysctl)

Tuning Time!

After lots of research and digging, mainly into these articles mentioned below, I was able to make some educated guesses on possible performance improvements. (If you know something better please reach out to me!)

After lots of trial and error these are the settings I landed on. Keep in mind, I’m running a virtual firewall with 8GBs of RAM and 16vCPUS.

These settings can be set under System > Settings > Tunables

hw.ibrs_disable=1

This is a CPU related tunable to mitigate a Spectre V2 vulnerability. A lot of people suggested that disabling it was helpful for performance.

net.isr.maxthreads=-1

This uncaps the amount of CPU’s which can be used for netisr processing. By default this aspect of the network stack on FreeBSD seems to be single threaded. This value of -1 for me resulted in 24 threads spawning (for my 24 CPU’s).

net.isr.bindthreads = 1

This binds each of the ISR threads to 1 CPU core, which makes sense to do since we are launching one per core. I’d guess that doing this will reduce interrupts.

net.isr.dispatch = deferred

Per this Github thread I linked earlier, it seems that changing this tunable to “deferred” or “hybrid” is required to make the other net.isr tunables do anything meaningful. So, I set mine to deferred.

net.inet.rss.enabled = 1

I decided to enable Receive Side Scaling. This didn’t come from the tuning guide either, it came from an OPNsense forum thread I linked earlier. In a nutshell, RSS is another feature to improve parallel processing of network traffic on multi-core systems.

net.inet.rss.bits = 6

This is a receive side scaling tunable from the same forum thread. I set it to 6 as it seems the optimal value is CPU cores divided by 4. I have 24 cores, so 24/4=6. Your value should be based on the number of CPU cores on your OPNsense virtual machine.

kern.ipc.maxsockbuf = 614400000

I grabbed this from the FreeBSD Network Performance Tuning Guide, this was their recommended value for if you have 100Gbps network adapters. The default value that came shipped with my OPNsense installation corresponded with the guide’s value for 2Gbps networking. I decided since I may want to expand in the future, I would increase this to this absurd level so I don’t have to deal with this again. You may want to set a more rational value, 16777216 should work for 10Gbps. The guide linked above goes into what this value does and other values it effects in great detail.

net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=65536
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536

These TCP buffer settings were taken from the FreeBSD Network Performance Tuning Guide, I didn’t look into them too deeply but they were all equivalent or larger buffers than what came shipped on OPNsense, so I rolled with it. The guide explains more about how these values can help improve performance.

net.inet.tcp.soreceive_stream = 1

Also from the tuning guide, this enables an optimized kernel socket interface which can significantly reduce the CPU impact of fast TCP streams.

net.pf.source_nodes_hashsize = 1048576

I grabbed this from the tuning guide as well, it likely didn’t help with my problem today, but it may prevent problems in the future. This increases the PF firewall hash table size to allow more connections in the table before performance deteriorates.

net.inet.tcp.mssdflt=1240
net.inet.tcp.abc_l_var=52

I grabbed these values from the tuning guide which are intended to improve efficiency while processing IP fragments. There are slightly more aggressive values you can set here too, but it seems these are the more safe values, so I went with them.

net.inet.tcp.minmss = 536

Another tuning guide value which I didn’t look into too heavily, but it configures the minimum segment size, or smallest payload of data which a single IPv4 TCP segment will agree to transmit, aimed at improving efficiency.

kern.random.fortuna.minpoolsize=128

This isn’t related to the network at all, but it was a value recommended by the tuning guide to improve the RNG entropy pool. Since I am doing VPN stuff on this system, I figure more RNG is better.

net.isr.defaultqlimit=2048

This value originated from my earlier linked Reddit thread, it was quickly added during the last batch of tunables that finally pushed me over the edge in terms of performance, and I decided I’d leave it even if it wasn’t doing anything meaningful. Increasing queuing values seems to have been a theme of the tuning overall.

—

☕ Like what you read? Did it help you?

Send some coffee and love https://buymeacoffee.com/truvis :)
Your support helps pay for licenses, research & development, and other costs that allow me to bring you new guides and content!

❗If you are new to my content, be sure to follow/connect with me on all my other socials for new ideas and solutions to complicated real world problems and jump start your career! New content drops daily/weekly along with tips and tricks :)

👉 W: https://truv.is
👉 T: https://twitter.com/thattechkitten
👉 Y: https://www.youtube.com/@TRUValueInformationSecurity
👉 G: https://github.com/truvis
👉 L: https://www.linkedin.com/in/truvisthornton
👉 M: https://medium.com/@truvis.thornton

OpnSense Firewall Configuration: Performance Tuning for Multi-Gigabit Internet and better speeds in general

Hardware offload good or bad? Trash?

OPNsense Tunables (sysctl)

☕ Like what you read? Did it help you?

Written by Truvis Thornton