My application depends on a third-party service for some features. Sometimes, there was packet loss between my application server and the third-party server. When such packet loss happens, HttpClient timed out per 7 seconds, although I set the connection timeout to 10 seconds. Why 7 seconds? In this post, I tell you how I find out about this unexpected timeout.
My application raised connection timeout earlier then I set. What happened?
Let’s first see how TCP works. The Client sends an SYN segment to open a TCP connection. My application couldn’t get the SYN+ACK Segment from the server. What happens if this starting SYN segment keeps getting lost in the network?
When the first SYN is sent and can’t get SYN+ACT, OS retries the SYN packet. How many times does the OS retry? It is different by OS. In Linux, you can check the value with the below command.
cat /proc/sys/net/ipv4/tcp_syn_retriesThe tcp_syn_retries indicates the maximum number of retries that the OS will try. If the value is 3 (default in Linux), OS retries three times when the first SYN packet doesn’t respond. The next question is how long it takes to finish retiring.
OS retries SYN packet tcp_syn_retries times (in Linux)
The OS can’t know whether the first SYN packet is lost or delayed. So it needs to have an algorithm for when to retry SYN. Let’s see the below diagram. How long should the Client wait before resending the SYN packet?
This time-out is called RTO, which is a retransmission timeout. Interestingly, I observed that this timeout is changed throughout each retransmission. Let’s see the below wireshark screen.
Please look at the Time column. The first retransmission is sent in 1 second. But the second retransmission is sent in 3 seconds, waiting 2 seconds after the first retransmission packets. I draw the above situation below.
When I change the tcp_syn_retries value higher than 2, It seems the retransmission timeout doubles its value for each retry.
TCP RTO doubles its value; 1, 2, 4, 8, 16, …
This RTO doubling rule is written in RFC. RFC 6298.
So My application OS raised TCP I/O exception after 7 seconds because tcp_syn_retries was set to 2, and RTO doubles its value for each retransmission!
I want my application to wait for a connection timeout of 10 seconds. So I raised tcp_syn_retries to three, which is calculated as 12 seconds maximum OS retransmission wait. Now my application can raise connection timeout as expected.
Testing Tips
I tested OS connection timeout using nc (netcat) and special IP 192.0.2.1. The nc is a tool like telnet. You can use this tool to connect TCP connection very easily. 192.0.2.1 IP is a reserved IP address, not a public IP address, and cannot be directly accessed on the Internet. So a TCP SYN packet always does not reach this IP address.
nc -4 192.0.2.1 80When the OS reached maximum retransmission (tcp_syn_retries), the above command will exist. If you want to see TCP retransmission packets in detail, you can use tcpdump and Wireshark.
sudo tcpdump -i any -A -vvv host 192.0.2.1 -w packet_dump.pcapI used the above tcpdump command and opened packet_dump.pcap with Wireshark.
To change values like tcp_syn_retries, you can type the below command.
sudo vi /etc/sysctl.confAnd add the below line.
net.ipv4.tcp_syn_retries = <new_value>To apply the above modification, you need to run an additional command.
sudo sysctl -pThen you can check the changed value like the first one I mentioned.
cat /proc/sys/net/ipv4/tcp_syn_retriesSummary
I introduced how the TCP SYN packet is retried at the OS level. The key points are how many times does the OS retry TCP SYN packets and how long the OS wait before the next retransmission. The first one is determined by tcy_syn_retries. The default value in Linux is three at the time of writing this post. The second is determined by RFC, the first RTO is 1 second, and it is doubled by each retransmission. (I couldn’t find how to change this algorithm or initRTO).
I hope this will help you understand the TCP retry mechanism at the OS level. Thanks for reading my post.
