Python in Production: server

The aim of this guide (I plan to create couple of separate parts for different aspects of running web application in production mode) is to introduce you to key concepts and problems that may occur on live server. I’ll touch server itself, database, application server and deployment and provide You with configuration files and flow that I use and rely on. Note that it’s extremely subjective, but I think that configuration may be considered at least as “reasonable default”. Let’s dig in, shall we?

Stack

Most of the posts reffer to the stack I work on, which is:

  • nginx
  • haproxy
  • uwsgi
  • redis
  • postgresql
  • python

Linux

By Linux I refer to Ubuntu since that’s the distribution I use for most of my projects. So, playing with linux configuration we can affect three things mostly:

  • I/O
  • Nginx
  • Memory

Max open files

By default set to 1024. Since sockets are used to communication between different tools, they affect conncurrent connections which our stack is able to handle. If we expect high traffic, it’s a good practice to tune that setting:

$ ulimit -n 99999
$ vi /etc/security/limits.conf 
nginx soft nofile 99999
nginx hard nofile 99999

Now reload the changes

$ sudo sysctl -p

Kernel queue for accepting new connections

By default set to 128, it represents a size of kernel queue for accepting new connections.

$ sysctl -w net.core.somaxconn=99999 
$ vi /etc/sysctl.d/haproxy-tuning.conf
net.core.somaxconn=99999

Usable ports

By default set to 32768–61000, it represents range of ports that can be used by our system. The number affects number of concurrent open connections.

$ sysctl -w net.ipv4.ip_local_port_range=”10000 61000"
$ vi /etc/sysctl.d/haproxy-tuning.conf
net.ipv4.ip_local_port_range=10000 61000

Socket recycling

DO NOT ENABLE IT !

Common misconception is to enable fast recycling (most of tuning guides provide such advice), so sockets do not stay in TIME_WAIT, like that:

$ vi /etc/sysctl.conf net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1

However, as explained here: Click its highly discouraged.

Filesystem access

In order to improve I/O we can tell linux not to store information about last file access or read time (which it keeps by default). In order to change that, modify confiration of a partition which your files reside in

replace

$ vi /etc/fstab 
UUID=<UUID> / ext4 errors=remount-ro 0 1

with

$ vi /etc/fstab 
UUID=<UUID> / ext4 noatime,nodiratime,errors=remount-ro 0 1

noatime affects files, and nodiratime directories respectively

In memory filesystem for /tmp

Putting

$ vi /etc/fstab 
tmpfs /tmp tmpfs defaults,nosuid,noatime 0 0

to your /etc/fstab file results in replacing a filesystem for /tmp directory with an in-memory filesystem. This will highly increase I/O performance on file uploads. Note that it may be a bottleneck when files that are being uploaded are large or if you are lacking RAM.

At the very end mount new filesystem:

$ mount /tmp

Getting swap right

When you’re forced to add some swap for your system be sure to put those two lines in your sysctl.conf:

$ vi /etc/sysctl.conf 
vm.swappiness=10
vm.vfs_cache_pressure = 50

which respectively tell our system not to swap data of RAM to swap place that often (swapiness) and tell our system to cache access data so it’s not looked up frequently (vfs_cache_pressure).

The two things I want you to remember after this part are:

  • Default configuration of you system is good, but may not be properly tuned for high loads and getting maximum out of tools from your stack (nginx, haproxy).
  • No one knows all that stuff by heart (at least I don’t), so if you find that article useful, save it somewhere so you can look it up later when it comes to configuration, or create an ansible playbook that deals with that stuff :-).

Cheers!


Originally published at jnosal.logdown.com.

A single golf clap? Or a long standing ovation?

By clapping more or less, you can signal to us which stories really stand out.