High Throughput with HTTP Connections

Written by Divya Verma

In this post, we will try to understand how can we achieve high throughput with HTTP Connections and how we deal with HTTP Connections in Linux systems.

We uncovered this when we faced HttpHostConnectException. This is basically the TCP Port exhaustion problem we encounter when we want to achieve high throughput in a system.

This post will focus on the methods by which we can deal with TCP Port exhaustion.

Context:

When we talk about scalable systems there are many things we need to see one of them being — how many outgoing connections can our application handle?

Problem:

We were getting this HttpHostConnectException in our system when we were receiving a large load of traffic; however, at all other times, we didn’t see any issues.

We did a significant number of load tests to understand what was actually happening, and then we noticed we are facing this issue when there are had more than 28k connections open. Now you might think, 28k is a huge number, why would a single host create those many connections, there must be something wrong with the code? So, to understand this we needed to understand how the HTTP Connection works.

SYSCTL Local Port Range and Orphaned Sockets

Port exhaustion is a problem that will cause TCP communications with other machines over the network to fail. Most of the time there is a single process that leads to this problem and restarting it will fix the issue, temporarily. But it can come back unless we understand the root cause of why we have so many ports open.

Before getting further let us understand what constitutes a TCP Connection and what inbound and outbound connection means.

In the majority of the cases whenever we talk about TCP connections and high scalability and the ability to support concurrent connections, we usually refer to the number of inbound connections (i.e the incoming connection request to the server).

For example, let’s say my application is listing to port 8080 for new inbound connections. When we say we can support X number of concurrent connections, we mean that all these connections are established on port 8080 on the server host.

While it is an incoming request for this application, there has to be a respective outbound connection from the client. For any client to communicate over the network, it must initiate an outbound connection.

When a connection is established over TCP, a socket is created on both the local(client) and the remote(server) host. These sockets are then connected to create a socket pair, which is described by a unique quadruple consisting of the local IP address and port along with the remote IP address and port.

Now, if you understand the quadruple, let us consider one client host and one server host, the client IP, server IP, and server Port will remain constant for any request made. So the only variable part to create unique connections is the client Port.

Now, this port is selected randomly on the source from the ephemeral port range and this port gets freed up once the connection is destroyed. That’s why such ports are called ephemeral ports.

$cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

By default, the total number of ephemeral ports available is around 28k.

Now you might think this 28k is a huge number and what is the system doing making 28k concurrent connections. To understand this we need to understand the lifecycle of TCP Connections.

SYN_SENT → SYN_RECV → ESTABLISHED once in an established state the TCP connection is now active. However, once the connection is closed post the communication it doesn’t immediately become available.

The connection enters a state known as the TIME_WAIT state for a period of 60 seconds before it is finally terminated. This is a kernel-level setting that exists to allow any delayed or out-of-order packets to be ignored by the network.

$cat /proc/sys/net/ipv4/tcp_fin_timeout
60

If you do the math, it won’t take more than 460 concurrent connections per second before the supposedly large limit of 28000 ephemeral ports on the system is reached, which is quite easy to be reached in a system that experiences high load.

When a connection enters the TIME_WAIT state, it is known as an orphaned socket because the TCP socket, in this case, is not helped by any socket descriptor but is still held by the system for the designated time i.e. 60 seconds by default.

Now let’s talk about how we can detect this?

There are several commands that are commonly used to analyze the TCP connections.

Socket Statistics(ss)

ss is the popular and faster replacement of Netstat where it directly fetches data from the kernel space. The options available with ss can be seen at

https://www.tecmint.com/ss-command-examples-in-linux/

The ss -s command shows all the TCP connections established on the machine. If this number increases above 28k, it is likely that we can get into such an issue. Note: This number can be higher than 28k if there are more than 1 service running on the machine on different ports.

Netsatat

The netstat command is the popular command that provides information about all sorts of connections established on the machine.

sudo netstat -anptl

This will show all the TCP connections established on the machine. The details include:

• local address/port
• remote address/port
• process id initiating the connection
• connection state

We can use this to grep on a particular outbound server if it creates more than 28k connections which can give us insight into the port exhaustion problem.

In the example above we can see there are numerous connections established to port 8080, as we can clearly see there are different source ports.

Note: This will show up connections in all-state.

We can see the number of connections modifying the above command as

netstat -antpl| grep 8080 | wc -l

For this example, the number is quite low but if we see it going above 28k it is likely that we are getting a port exhaustion issue.

Now let's look at all are the possible solutions:

Connection Pool

So the number of connections that your process is creating is dependent on how many connections you have configured in your application to create.

So on careful analysis, we can use the response time and the number of ports that we can use (i.e. 28k) to identify what is the maximum number of threads we can configure.

For example, if your outbound connection response time is around 200ms, you can, at most, have 93 threads. Based on the calculation, one thread will use 5 ports in a second, in 60 seconds around 300 socket ports, and 28k/300 would be around 93 threads.

So if your application can sustain with this thread calculation you can use this number and Yay! Problem Solved.

Additional hosts to serve request

Having a number for the threads from the above calculation, some more hosts could be added so as to sustain the load. But that could incur more cost and also if your application is not so CPU intensive you will end up wasting CPU and memory capacity of the host as you are probably not using each hosts to its fullest.

TCP Kernel Parameters

Now let’s discuss how can it be solved by changing some Kernel parameters that affect TCP connections. These options discussed below are used by the Kernel to create/maintain TCP connections.

First, let’s look at the port range in the Port Range Exhaustion Problem.

We already saw that the default number of ports is quite low at around 28k, so we can try and increase that to max of 64k, as kernel only reserves 1024 ports for select services, the rest can be made use of.

echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range

Though this doesn’t serve as a fool-proof solution it just increased the limit from 28k to 64k but it will nonetheless provide us with some breathing space.

Does this mean I can only get about 64k concurrent connections from a single client machine? The answer is NO.

In this scenario, both client process are connecting to 2 different server/proxies so we should be able to create a max of 120k connections.

To understand how we can get 120k connections, we need to see the 4 tuple (client IP, client Port, server IP, server Port) that forms the connections, if any of them change we have new connections. Now how this works is the same port being used by two client processes? This is resolved using file descriptors, every connection basically creates a file descriptor, and the number that limits no. of connections will be the max file descriptor configured.

Second, let’s look at if we can reduce the TIME_WAIT time

As we saw earlier, the connection once it is complete, stays in time_wait for the delayed packets for 60s by default. If we can somehow reduce this, that could solve a few problems for us.

For e.g.: If we have a p99 of below 10ms it doesn’t make sense to wait for the packets to get delivered for 60s. So, in such case I would consider reducing this.

echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
or 
--> vim /etc/sysctl.conf
--> add the following line at the end of the file
net.ipv4.tcp_fin_timeout = 15
--> Reload the sysctl.conf 
sysctl -p

With a command similar to above the port will wait only for 15s in the TIME_WAIT state.

Again this might not be a fool-proof solution but it will surely give us more room and scalability that can be achieved from each host.

Third, let’s look at if we can reuse the ports in TIME_WAIT state

Another simple solution is to enable a Linux TCP option called tcp_tw_reuse. This option enables the Linux kernel to reclaim a connection slot which is in TIME_WAIT state and reassign it to a new connection.

echo 15 > /proc/sys/net/ipv4/tcp_tw_reuse
or 
--> vim /etc/sysctl.conf
--> add the following line at the end of the file
net.ipv4.tcp_tw_reuse = 1
--> Reload the sysctl.conf 
sysctl -p

Other solution

We can have other solutions where we can make the server run on say 2 different ports wherein we can make double the number of connections.

Conclusion

Lastly, there is no single solution for such a problem but a combination of them that can let us achieve the kind of scale we want to.