How to deal with “Too Many Open Files” log-storm

Common Scenario for Network Connection Timed out Error

Photo by Anandu Vinod on Unsplash

Introduction and Background:

You are in trouble if your HTTP server greets you with a 599:Network Connection Timed out Error. The first thing to check is if the server is reachable and you can SSH into it. Then comes to the rescue, the log file of the server. They would most likely look something like this.

HTTP: Accept error: accept tcp [::]:<port_number>: accept4: too many open files

One might wonder, what has too many open files got to deal with HTTP accept error. The reasoning behind this is quite interesting. Every socket in Debian is treated as an open file. That’s how they are related! Since every socket is an open file and the server has exceeded the limit of the number of open files, the server is not able to accept new TCP connections.

# command that counts the number of open files by the service.
lsof -u <user-running-the-service> | wc -l

If it is a production server and you are not sure what the deal is, except for the fact that you realized that there are too many open connections, the quick and dirty fix is to restart the service. Since it is a TCP server all the existing connections would tear down and the…