Case study: Network bottlenecks on a Linux server: Part 2— The Kernel
In the previous article I went over some potential bottlenecks related to the NIC itself. In this article, we’ll be looking at some of the kernel tweaks that can be useful, and the effect it may have on the network traffic.
The series is divided into (likely) these four parts:
- Part 1: The NIC
- Part 2: The Kernel (this article)
- Part 3: Interrupts
- Part 4: Going further
In the first part we saw what the NIC ring buffer was and how it could be tweaked. In our case, those tweaks solved only the first part our issue, and when fixed exposed another bottleneck further down (up?) the OSI model. We still had a bottleneck somewhere, so the next question was, “so what happens to our packets after the NIC?”
The observation
After the NIC issues that we looked at in the previous article, the next observation made was a correlation between the connection drop off cliff during peak time, and the increase in packets reported as being “collapsed” and “pruned.” The collapsing usually started increasing an hour or so before the pruning. When tracked by Grafana it looked something like the following for collapsed packets: