Deciding on a language for your web service
At Singularity, we primarily use Python for our web services. On the frontend, we use Typescript. We also use Elixir for our real-time push server. We recently started a project that requires very high I/O throughput and were concerned that Python may not be the best tool to serve us in that use case. So we asked the question:
Which language should we use for a high-throughput web service?
Based on our in-house experience on our team, we had a few candidate languages that we wanted to benchmark. As a team we decided to try testing:
- NodeJS
- Java
- Rust
- C++
We decided to devote about 2 hours to standing up each language with a framework that seems to have a large community support and thorough documentation. The web servers would each interact with the same database in a docker-compose network with the PostgreSQL DB. The web servers would have 2 endpoints that only do: fetch item by ID, paginate list of items with page/offset parameters.
The Testing
The languages and frameworks we actually tested were:
- Python (Flask/SQLAlchemy/psycopg2)
- NodeJS (express/TypeORM)
- Rust (Actix Web/tokio_postgres)
- Async Python (FastAPI/SQLAlchemy/asyncpg)
- Elixir (Phoenix/Ecto)
Although we had agreed to also try out C++ and Java, I had trouble setting them up, even for a simple case. Researching them took far too long, lacked (public) resources and community support that I could find, and was unable to get a JSON API → PostgreSQL db service working in a modest amount of time. I’m open to trying again with those at another time, but the difficulty in approaching them turned me off of them for this set of tests.
For all of the frameworks and languages listed, I followed the docs’ recommended practices for creating a production-ready docker image. I then ran those images as containers in a docker network on my laptop that also has a PostgreSQL 16 docker container running, so they all hit the same database. I ran the tests using Locust locally.
This article will not dive into the why of each result set, that would make the research and article much longer. Additionally, the point of this exercise is to see what can be learned the quickest for the highest IO throughput of a server.
For each round of results, we’ll go through the number of concurrent users and how the different frameworks handled them.
Under 1,000 users
A set of tests for 1, 10, and 100 concurrent users that each send up to 2 requests per second.
Python
NodeJS
Elixir
Rust
Async Python
For these set of tests, we see what we would expect. Almost all setups handle the ~200 concurrent requests just as they come in and are served. The only exception is Python using the async/await syntax. Unfortunately, it had an error rate of about 50%. When looking at the error that was in the logs, it appeared as:
sorry, too many clients already
Propagated from the asyncpg driver. Looking this error up shows that it came from the PostgreSQL db directly and was not an error in application code. I’m sure that there could be some tweaks and configuration changes that I could do to make this setup work well, but that would’ve defeated the purpose of the exercise.
All subsequent tests will exclude the Async Python setup.
1,000 Users
Python
NodeJS
Elixir
Rust
Here, we are asking our setups to keep up with roughly 2,000 requests per second. All frameworks make pretty light work of it except Python. Why can’t Python keep up? Why does it cap at around ~200 reqs/s? What is my exact setup?
All good questions, I’m curious too, but not the point of the exercise. However, since we didn’t see errors happen, I’ll continue to include Python in the last set of tests.
2,000 Users
Python
NodeJS
Elixir
Rust
Here, we are asking our setups to keep up with roughly 4,000 requests per second. All frameworks that try to reach that level clearly have a tough time keeping up. However, because every setup (excluding Python) seems to have very similar behavior, I’m tempted to believe that there is a bottleneck on the laptop I’m using to run the tests and not a bottleneck with the setup itself. That may or may not be true.
Conclusion
For workloads with high throughput, where IO is the bottleneck, it appears clear that you won’t go wrong choosing and learning any of these setups aside from Python.
I won’t dissuade anybody from using Python, we have used it for years, enjoy its ease-of-setup and syntax, and will continue to use it, especially for data analysis and manipulation workloads.
However, we did this test because we will soon have a new product with a different workload than our previous products. We wanted to see if we should switch up our stack to better serve the new product.
Curious what the new product is? Interested in our stack? Disagree with the results?
We’re hiring a software engineer! Email us and tell us your thoughts, we’d love to hear from you 🌱
