Scaling to a Millions WebSocket concurrent connections at Spoon Radio

Authors: Edward J. Yoon, Yeontaek Kim, Kyutaek Oh, Ellie Kang, Caley Baek

We have been working on developing Heimdallr, our chatting platform for Spoon Radio, along with a supporting server-side infrastructure that can handle millions of concurrently connected devices.

Heimdallr’s infrastructure maintains a persistent socket connection to our cloud, which lets us push out messages to devices in real-time. This brings together all the complexity of handling hundreds of thousands of persistent socket connections server side, managing the connection on the client side as users pass in and out of coverage or switch between networks, and detecting and closing half-open sockets as quickly as possible.

In this article, we describes how we handle massively concurrent WebSocket connections using Heimdallr on AWS clouds.

A TCP Port field is 2x bytes and holds a quantity of 65536. This number limits the amount of addresses a server can have, but does not limit the number of clients to ~64k. Each TCP Packet has two Port fields one for the destination and one for the source (as well as two IP addresses). In Kernel options, several parameters exist to allow for tuning and tweaking of socket-related parameters. In /etc/security/limits.conf file, we have to set the “soft” and “hard” nofile limit to 1000000. Instead of using the “ulimit -n” as some people do we have to, for some reason, specify the “soft” and “hard” limits for both root and all other users separately.

* soft nofile 1000000
* hard nofile 1000000

In /etc/sysctl.conf there are few options we have modified. First is the fs.file-max, the maximum file descriptor limit. The default is quite low so this should be adjusted. Be careful if you are not ready to go super high. Second, we have the socket buffer parameters net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. These are the buffers for reads and writes respectively. Each requires three integer inputs: min, default, and max. These each correspond to the number of bytes that may be buffered for a socket. Set these low with a tolerance max to reduce the amount of ram used for each socket. The relevant portions of our config look like this:

fs.file-max = 1000000 // the maximum file handles that can be allocated
fs.nr_open = 1000000
net.core.somaxconn=1000000
net.core.netdev_max_backlog=1000000
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.nf_conntrack_max=1048576 // if doesn’t work, execute ‘$]modprobe ip_conntrack’ ( on ubuntu )
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.ip_forward=1
net.ipv4.conf.ens3.forwarding=1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_tw_reuse=1

Once the configuration is finished, you have to apply the changes with the following command: ‘sysctl -p’

We use HAProxy as a WebSocket Load Balancer. We tried to use latest Apache 2 with ws_tunnel module for Load Balancing, but it was not good for scaling. First of all, to increase the open file limit of HAProxy processors, we have to set the LimitNOFILE to 100000 or infinity in /etc/systemd/system/multi-user.target.wants/haproxy.service file like below:

[Service]
LimitNOFILE=1000000
…

Another big one is a port exhaustion problem. To avoid TCP Port exhaustion problem that causes TCP communications with other machines over the network to fail, we have to use multiple virtual network interfaces.

‘sudo ifconfig ens3:1 10.0.11.32 netmask 255.255.255.0 up’,

This command bind a intranet address to a virtual network interface ens3:1 whose hardware interface is ens3. This command can be executed several times to add arbitrary number of virtual network interfaces.

Next, we need to config HAProxy to use these fresh IPs. There is a source command that can be used either in a backend scope or as an argument of server command. This is how HAProxy config file looks like:

backend bk_web
#log global
 balance roundrobin
server heimdallr1 10.0.1.11:8080 maxconn 100000 weight 10 cookie heimdallr1 check source 10.0.11.32
server heimdallr2 10.0.1.12:8080 maxconn 100000 weight 10 cookie heimdallr1 check source 10.0.11.33
server heimdallr3 10.0.2.11:8080 maxconn 100000 weight 10 cookie heimdallr1 check source 10.0.11.34
server heimdallr4 10.0.2.12:8080 maxconn 100000 weight 10 cookie heimdallr1 check source 10.0.11.35

We use Tsung for WebSocket concurrency load tests. In our tests on AWS 4 nodes Heimdallr cluster, we had check that 100k concurrent connections can be established.

Heimdallr is currently open sourced under the Apache License Version 2.0 and is available at https://github.com/edwardyoon/Heimdallr . Enjoy!

Originally published at medium.com on November 12, 2018.