Running 100K concurrent connections on a single commodity hardware server

During Ind Vs Nz Cricket match 2019 World cup, Hotstar scaled to 25M and I particularly found it fascinating how they implemented the broadcast of commentary messages to Millions of live users. At the same time I read articles of how WhatsApp holds millions of concurrent user connections using high end servers. Recently just for fun I wanted to test run a messaging server capable of at least 100K concurrent users on a single commodity hardware server.

Server Application

It’s a simple broadcast messaging application using the Elixir’s Phoenix framework. Last year I learned Elixir, which is a sugarcoat on top of Erlang and since WhatsApp used Erlang I wanted to use the same.

Client Application

I wrote a sample Java client using Java Phoenix Client which creates a persistent connection to this Server and sends some random messages after every some random duration. I ran this client on a Kubernetes cluster where I was running 2000 users per pod. I used GKE to run a cluster with the deployment of 50 nodes.

Server Hardware

8 core 16 GB RAM google cloud instance

Client hardware

2/4 CPU 8 GB RAM GKE general purpose machines on GKE cluster with 50 nodes

Challenges

I had to update File descriptor limits and other relevant ulimits on the server.
Also the Cloud usage quota limits were something which was really problematic to get around to run such a cluster.

Result

The server was able to hold the 100K / 0.1 Million connections without sweating with RAM usage of around 3GB and some 20–30 odd percent of CPU consumption.

Why is Elixir able to do it ?

Because of OTP. OTP (Open Telecom Platform) is an awesome set of tools and libraries that Elixir inherits from Erlang. OTP contains a lot of stuff, but OTP in the context of Elixir, we usually mean the Erlang actor model that is based on lightweight processes and is the basis of what makes Elixir so efficient. The concurrency model relies on Actors, a contained process that communicates with other processes through message passing.

References

https://elixir-lang.org/
https://elixircasts.io/
https://www.phoenixframework.org/
https://serokell.io/blog/elixir-otp-guide