note: I am the owner of bhop.rip, a competitive bhop server hosted in Australia, you can talk to us at https://discord.gg/KyM8qVg
note 2: most of these things should transfer to CS:GO servers or pretty much any source game for that matter, but I have no experience in other games so I cannot make any recommendations.
Introduction
Hello, thanks for looking at my article. I have been in the “bhop” scene for about 10+ years now and I started hosting my own bhop servers around 2014 when all the popular Australian servers (AusMP, Imm-Net, DTK, Mr.#) at the time either died or went unmaintained. I’m writing this article since I’ve accumulated a bunch of specific knowledge over time and it feels wrong to keep it all to myself.
Traditionally people would purchase dedicated servers running Windows because of the need to run a particular extension to SRCDS (source dedicated server executable) at 100 tick rate. However this functionality has since been ported to GNU/Linux so we are not longer restricted to relying on the NT kernel. You can read more about tick and other mechanics at http://flafla2.github.io/2015/02/14/bunnyhop.html.
In this article I will cover the minimum system requirements and kernel/system configuration for optimal game server performance. By optimal performance I mean the least chance of any packet drop, jank, lag, jitter — whatever you want to call it.
Generally speaking it is not proper to expect clients with over 60–80ms ping to be able to bhop correctly on your server so we are not designing the system to handle anything higher than that.
Hosting Provider Selection
The first thing you will need is a dedicated server, this is not really negotiable and the main reason for that is that if you are using any sort of VM hosting (Xen, OpenVZ, KVM, VMWare, etc) you will not be getting dedicated CPU time nor will you have control over the CPU’s scheduling algorithm. Our aim is to end up with a dedicated CPU core that runs only the dedicated server process and is not interrupted by anything else.
When you are using virtualization the CPU is split between running the host computer’s processes (unless they have configured isolated CPU’s) but you are also contending for CPU time with other virtual machines(!). If you want to see it for yourself, try making a new EC2 instance and look at the “guest time” and “steal time” parameters in a tool like top or vmstat.
Aside from the need for a dedicated server, you will also need to pick a provider with decent networking, as not all networks were made equal, and the heavens do not favour a bhopper that needs to travel through many hops or misconfigured networks. The key factors you should be looking for are:
- Dedicated network port (not shared)
- 1GbE speed with at least 10TB of bandwidth (lower suggests a reseller or generally less-equipped provider)
- At least 3 major network providers (most good hosts will advertise this)
- 500Gbps+ DDOS protection/filtering
In my experience, G-Core (http://g-core.lu) has been extremely good. I have been using their services for a few years now and they have had almost zero downtime and the packet latency is very consistent.
You should try and avoid cheap providers like OVH as they host many “undesirable” services which attract large network attacks and they will not provide good support unless there is a hardware issue. The other issue is that hosts like OVH will often have (and boast) a large, global network but they will mostly rely on their own internal routes as a cost-saving measure which results in worse congestion and unpredictable latency (which is extremely detrimental for a bhop server).
The next selection factor will be the CPU. We want the srcds process (and threads) to run uninterrupted on a single CPU core. Depicted below you can see that there are some Sourcemod threads running, these need to be on the same core as they affect the in-game environment. For that reason, it is also optimal to avoid having too many plugins loaded, as each extra one can incur minute delays if they hook to things like the physics or movement code.
The main things you need to consider for the CPU are:
- At least 2 cores (given that you will usually run a MySQL database in conjunction with srcds+sourcemod you should aim for 4 cores)
- At least a 3.5 GHz base clock speed. A lot of bad providers are currently offering cheap Xeon-D processors with a 2.6 Ghz base clock speed but this is not a good idea as we are aiming to support a server with at least 16 players and 2.6 GHz will not cut it.
The srcds binary is rarely updated and it is unlikely that you will benefit from having a latest generation CPU since there has been no major innovation in the past 6 years for CPU design.
RAM does not really need a section here, just make sure you have at least 8GB of DDR3–2400 or newer.
The last thing you will need is an SSD or RAID setup for storage as you will find that running a server with a lot of maps will incur a startup performance and map change speed penalty from re-scanning the map directories. At the time of writing this, there were about 4600 bhop maps in our local server with about 1300 of them being zoned and in the map cycle. If you extrapolate that to a single enterprise-grade HDD you can see how it will not perform.
Using a Custom Kernel
The two main objectives of this guide are to setup a tickless kernel and to setup cpu isolation so we can run srcds on one core. Technically speaking you could just do core isolation and net the most benefit however this setup is intended to be the best possible for your users.
Debian is the preferred distribution for making a custom kernel as it provides a very simple interface for building your own kernel into a .deb package which automatically works with grub, initrd, and other boot stuff.
The current official guide on building a custom “Debian” kernel can be found here: https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-building
Essentially the process for this is:
1. Get the kernel source and extract it to a reasonable location (/usr/src)
3. Go to the folder and edit the parameters with make nconfig
4. Run the build (make clean && make deb-pkg)
5. Install it with dpkg -i
There are a number of ways to configure the kernel but we will just focus on editing it through the nconfig system. Once you are in the nconfig menus, you need to set the following options:
General Setup -> Timers Subsystem -> Timer tick handling -> [*] Full dynticks system (tickless)General Setup -> Timers Subsystem -> [*] High Resolution Timer SupportProcessor type and features -> Processor family > [*] Core 2/newer XeonProcessor type and features -> Preemption Model -> [*] Preemptible Kernel (Low-Latency Desktop)Processor type and features -> Timer frequency -> [*] 1000 HZ
If you are interested in the nitty-gritty details of these settings, there is a good guide here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/timers/NO_HZ.txt
I will not be covering building and installing the kernel in this guide as the Debian handbook describes it better than I ever will.
Using Dedicated Cores
Assuming you followed the above instructions, you now have a shiny new tickless system! which is great but your processes are still running on all cores, so you need to pick a core for the srcds process to run uninterrupted. For the sake of simplicity, this guide will show you how to isolate core 3 (cores are counted from 0).
By default the Linux kernel will use all cores when you boot, however you can easily change this by setting the isolcpus boot parameter. On a Debian system this is easily changed by editing /etc/default/grub and appending isolcpus=3to the GRUB_CMDLINE_LINUX_DEFAULT value after quiet
After you do this you need to run update-grub to make the changes apply to GRUB’s boot.cfg
After you reboot the system, you will notice the absence of activity on core 3. This indicates the kernel is definitely not scheduling processes to run on the core. But wait.. we forgot about hyper-threading! you need to run
echo 0 > /sys/devices/system/cpu/cpu7/online
This will kill the extra “core” dead, for sure.. maybe.
If you are really interested in whether it actually was the right core you can inspect /sys/devices/system/cpu/cpu3/topology/thread_siblings_list and see if that was the right one. If it wasn’t then you have a really messed up CPU.
The last thing you need to do is use taskset when you are launching the srcds_linux process, thankfully this is as simple as running:
taskset -c 3 /my/location/to/srcds_linux <arguments>
You should also renice the process to -20 since it will be the only thing running on the core. You can use renice to achieve this or you can just open topor htop and do it from there with <r> or <F7> respectively.
I use a script to automate this whole process but I haven’t put it up online. If you are really lazy you could ask me for it and I might give it to you.
SRCDS Tuning
Wow! look at you with your fancy pants dedicated server!
By this point in time you’ve probably lost attention so I will just dump the configuration options you should set in cstrike/cfg/server.cfg:
host_framerate 0
fps_max 300
g_ragdoll_maxcount 0
net_compresspackets 0
net_maxcleartime 0.0001
sv_competitive_minspec 1
sv_client_min_interp_ratio 0
sv_maxrate 0
sv_minrate 300000
sv_mincmdrate 100
sv_minupdaterate 100
sv_maxcmdrate 100
sv_maxupdaterate 100If you are interested in what these parameters do, you can have a look at https://developer.valvesoftware.com/wiki/List_of_CS:S_Cvars but basically we are just setting the server to have the correct update rate for clients and forcing a high minimum rate for smooth gameplay. When we set such a high minimum rate it will affect people with high ping as they will usually not be able to get reliable continuous packet flow to our server. Too bad.
Oh yeah, you definitely need to be running the server at 100 tick, in case you can’t find the library, it’s right here: https://github.com/daemon32/tickrate_enabler
Further Tuning Options
If you got this far, congratulations! now we will see what the final bits of performance we can eek out of the system. The benefits of changing these things are mostly superficial but it could be argued that stripping a couple of milliseconds here and there will make all the difference. So let’s have a look.
First, we are going to tell irqbalance (you have that installed, right?) to not distribute IRQs to core 3 by adding the line below to /etc/default/irqbalance
FOLLOW_ISOLCPUS=yesThe reason why we are doing this is because by default Linux will handle interrupts from devices (i.e. your network card) across all available cores.
Next, we will adjust /etc/sysctl.conf to use “sensible” settings for a game server:
net.core.rmem_default = 31457280
net.core.rmem_max = 12582912
net.core.wmem_default = 31457280
net.core.wmem_max = 12582912
net.core.somaxconn = 4096
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_low_latency = 1Run sysctl -p to propagate the changes. These changes are mostly just for increasing buffer sizes which may show slightly better performance for a heavily-used server. There are a few options for congestion control that you could set but I have not really thoroughly tested the available algorithms so I will not make a recommendation here.
Lastly, we will turn off the fancy options and fake lag for the NIC:
ethtool -C <adapter> adaptive-rx offethtool -C <adapter> rx-usecs 0 rx-frames 0 rx-usecs-high 0 rx-usecs-low 0 pkt-rate-low 0 pkt-rate-high 0ethtool -A <adapter> rx off tx off
Other things you could do if you want to feel smug:
- Set NOTRACK for the listening IP address in iptables
- Turn off all frequency scaling for your CPU in the kernel config
- Turn of all power management options in the kernel config
- Turn off HyperThreading in the BIOS
- Tune your NIC with other tools
By this point you’ve finished the platform setup, well done!
Recommended Timer/Sourcemod Plugins
Currently there are two timers which are up-to-date. They both have advantages and disadvantages, here is an overview:
Shavit’s Timer (https://github.com/shavitush/bhoptimer)
- Modern timer with useful features, segmented mode
- Does not have replay for bonus bots
- Lets you run multiple servers off the same database (via plugin)
- Is completely free and open source
Blacky’s Timer (https://github.com/blankbhop/btimes)
- Wants you to pay $200 USD for it
- Has no killer features
- Leaked by Blank so you don’t need to pay for it
- Some clans claim running this black-box plugin is an advantage but I don’t think it is smart to trust records from a system which can’t be inspected!
So really the choice is entirely yours to make but you’re gonna make the right choice, huh?
Other plugins which will complement your server nicely:
- Weapon Cleanup: https://forums.alliedmods.net/showthread.php?p=870661 (useful for preventing pointless lag from too many entities)
- ORYX: https://github.com/shavitush/Oryx-AC (the only useful anticheat)
- ljstats: https://forums.alliedmods.net/showthread.php?p=2060983
- rngfix: https://forums.alliedmods.net/showthread.php?t=310825
- SM Super Commands: https://forums.alliedmods.net/showthread.php?p=498802
Conclusion
This guide was partly a brain dump for me since I find I remember things better if I write them down somewhere. The main thing you should take from this guide is that buying from a slot-based game server provider is a pretty awful choice for latency-sensitive game modes like surf, trikz, and bhop. I mean after all, do you really want to share your CPU with 4 minecraft servers?? Do it yourself and enjoy a smoother gaming experience!
if you have any questions or criticism you can talk to me on steam: https://steamcommunity.com/id/merzbau
