Socket Timeout — An Important and Sometimes Complicated Issue with Python

During my experience working with Python, I’ve had several cases where a network client was left hanging while trying to request a server. After spending some time researching, the root cause was identified and it was found that the server was waiting for a response but there was silence (like being ghosted on a Tinder match).

[Photo on masculino.ru]

Of course, an operating system can raise a Connection timed out error, but it doesn’t always seem to happen on hang connections. If it did, probably wouldn’t even run into this issue on the job.

In order to avoid the annoying problem of waiting for an undetermined amount of time, sockets (which are responsible for network communication) support a timeout option which raises an error based on a time limit. Unfortunately, when developers develop their amazing network libraries, they may omit such non-obvious cases and forget to provide socket timeout settings, despite of the docs recommendation:

The connect() operation is also subject to the timeout setting, and in general it is recommended to call settimeout() before calling connect() or pass a timeout parameter to create_connection(). However, the system network stack may also return a connection timeout error of its own regardless of any Python socket timeout setting.

Let’s look at a simple example of how to simulate hang socket (at a minimum level in Mac OS):

# server side
import socket
server = socket.socket()
server.bind(('127.0.0.1', 30000))
# client side
import socket
client = socket.socket()
client.connect(('127.0.0.1', 30000))  # hang :(

At this moment, the client is hanging, waiting for the server to enable to connection accepting, like server.listen(1).

It won’t actually hang for eternity, and after some time should eventually raise a TimeoutError: [Errno 60] Operation timed out. This happens because the system function timed out at the system level with the error ETIMEDOUT. As mentioned above, it seems the ETIMEDOUT error doesn’t always happen or it could be that the timeout value can be obscenely large. In either case, it’s not controlled in your code. The variant which I will show has been much more reliable for me.

Before I go into showing issue solution, let’s go deep into Python socket implementation in order to understand why it hangs.

1. sock_connect is a C-function, which is called on client.connect. Its code is clear and we see that it calls ↓
2. internal_connect function, and this code is harder to understand, but I will give a hint — we are interesting in ↓
3. sock_call_ex function, which has an eternal cycle while (1) {, where socket communication happens and where it tries to wait for an established connection as well.
4. Also in sock_call_ex is a processes timeout option in if (has_timeout) {. If a timeout is achieved it returns the error: PyErr_SetString(socket_timeout, “timed out”);.

Let’s see how the timeout option helps with a hang socket:

# server side
import socket
server = socket.socket()
server.bind(('127.0.0.1', 30000))
# client side
import socket
client = socket.socket()
client.settimeout(3)
client.connect(('127.0.0.1', 30000))
----> 1 client.connect(('127.0.0.1', 30000))
timeout: timed out

As said above, this option can be omitted in libraries which use sockets. Fortunately, Python gives you a chance to set up socket timeout for all new sockets, which will be created during application work:

import socket
socket.setdefaulttimeout(10)
sock = socket.socket()
sock.timeout
10.0

If this would be enough, the article would finish here. Unfortunately, socket timeout can be reset with a sock.settimeout(None) that some libraries do rather rashly.

A solution for this is monkey-patching of the socket module, like this:

But in the socket there is another way to reset timeout:

import socket
socket.setdefaulttimeout(10)
sock = socket.socket()
sock.timeout
10.0
sock.setblocking(True)
sock.timeout
None

Let’s see what happens under the hood of this method:

s->sock_timeout = _PyTime_FromSeconds(block ? -1 : 0);

Here it changes the timeout, without worrying about its default value. Why it works this way you can read more about in the socket docs, but it’s easy to patch:

Checking results:

import socket
socket.setdefaulttimeout(10)
sock = socket.socket()
sock.timeout
10.0
sock.settimeout(20)
sock.timeout
20.0
sock.settimeout(None)  # resets to default timeout
sock.timeout
10.0
sock.settimeout(20)
sock.timeout
20.0
sock.setblocking(True)  # keeps existing timeout
sock.timeout
20.0
sock.setblocking(False)
sock.timeout
0.0
sock.setblocking(True)  # resets to default timeout
sock.timeout
10.0

No more hang sockets any longer.

Keep in mind that these socket monkey-patches may prevent developers from setting a socket blocking mode if the default timeout is set. Unfortunately some libraries do this thing unconsciously and you ultimately have to choose between canonical and working code.

[Photo on pafjimenez.wordpress.com]

If you’re interested in other Python articles, check out Weak Reference and Object Management within Python Threads, ML Code vs AWS Lambda Limits, or Encountering Some Python Trickery