We recently encountered a server problem that was a little more than just a configuration mishap, and we had to dig deeper. Here’s the story:
This week, we noticed that our virtualized Arch Linux servers were having strange network performance issues. While the upload speed was quite good, the download rate had decreased to about 200 KB/s and there was a noticeable amount of received packets being dropped.
First, we thought there was a network problem beyond our reach, but after booting a live system, everything was just fine. So the problem had to lie somewhere within our server setup. As we looked further into it, we found that the problem was introduced with kernel 4.12. More specifically, the performance issue seems to be related to changes done to the virtio net driver. We discovered this patch, which looks like it addresses the bug.
Of course, since this is an issue with virtio, only virtual machines are affected. Any kernel older than 4.12-rc1 doesn’t contain this bug.
We downgraded our Arch Linux installations to kernel 4.11.9–1 and will stick to 4.11 until the problem is fixed.
For Arch Linux specifically you can find information on how to downgrade the kernel here. See the Arch Linux Packages Archive for a list of all available kernel versions.
