How to analyze TCP dump for slow backends?
The TCP dump command-line packet sniffer tool allows you to capture or filter TCP/IP packets received or transferred across a network. It is compatible with Linux/Unix-based operating systems. You can use yum to install this utility as follows:
yum install tcpdump
# Must capture 3-way handshaking
window scaling: TCP window size is one of the most commonly used options for network troubleshooting and application baselines. The TCP window size, or TCP receiver window size as some refer to it, is simply an advertisement of how much data (in bytes) the receiving device is willing to receive at any given time. The receiving device can use this value to control data flow or as a flow control mechanism.
Some operating systems will calculate the maximum TCP window size as a multiple of their maximum segment size (MSS). For example, the default value in Microsoft Windows 2000 on Ethernet networks is 17,520 bytes or 12 MSS segments of 1,460 bytes each.
This packet has a scaling value of eight in the screenshot above, which is converted to 256. The calculation is simple: two to the factor (or power) of eight equals 256. Only during the TCP three-way handshake is the window scale option used. The window scale value is the number of bits used to shift the 16-bit window size field to the left. The window scale value can be changed between 0 (no shift) and 14.
# Delay in time difference and decrease in calculated window size
window scale: 8
TCP send buffers: The server provides the first layer of buffering between the server and the browser. The operating system keeps a TCP send buffer in which the server writes data. Once the data is in the buffer, the operating system is responsible for delivering it as needed (pulling from the buffer as data is sent and signaling to the server when the buffer needs more data). A large buffer reduces CPU load by reducing the amount of writing the server must do to the connection.
The actual size of the send buffer must be large enough to keep a copy of all data sent to the browser but not yet acknowledged in case a packet is dropped and some data must be retransmitted. A buffer that is too small will prevent the server from fully exploiting the client’s connection bandwidth (and is a common cause of slow downloads over long distances). In the case of HTTP/1.x (and many other protocols), data is delivered in bulk in a known order, and increasing the buffer size has no disadvantage other than increasing memory usage (trading off memory for CPU).
The issue with large send buffers in HTTP/2 is that they limit the server’s ability to adjust the data it is sending on a connection as high-priority responses become available. Once the response data is written into the TCP send buffer, it is no longer under the server’s control and is committed to be delivered in the order in which it is written.
The optimal send buffer size for HTTP/2 is the smallest amount of data required to fully utilize the browser’s available bandwidth (which is different for every connection and changes over time even for a single connection). In practice, the buffer should be slightly larger to allow for some time between when the server signaled that more data is required and when the server writes the extra data.
TCP Window Full flags usually indicate that the sender is using the full capacity of the TCP flow, which is limited by the recipient’s receive window. The network’s capacity may be increased. More information is provided below.
TCP Zero Window: When a TCP receiver’s buffer begins to fill, the receive window can be reduced. If the window is full, it can be reduced to zero, signaling the TCP sender to stop sending. This is known as “closing the window.” This usually means that the network is sending data faster than the receiver can process it.
# Time to live
TTL is the amount of time or “hops” that a packet is allowed to exist within a network before being discarded by a router. TTL is also used in other contexts, such as CDN and DNS caching. It is an 8-bit binary value set by the sending host in the Internet Protocol (IP) header. A TTL prevents data packets from being circulated in the network indefinitely. TTL has a maximum value of 255. Administrators can change the value of TTL from 1 to 255.
if TTL keeps reducing, that indicates we may have a bad route path and need to check the DNS to evaluate why we need to take a long path to reach the server.
TCP Delta Time measures the amount of time that has passed between the previous and current packets in a conversation. if delta-time increases then that will impact the ACK and the buffer, which may lead to performance issues as well.