The Startup
Published in

The Startup

Client-side Observations about Web Service Technologies: Using Apache Bench

Thundering Web Requests: Part 3

Setup

Node Setup

Benchmarking Tool Selection

Number of Concurrent Requests (Connections) Configuration

Network Traffic Configuration

Number of Random Integers (Compute Load)

Execution

Keeping the Raspberry Pis Cool

Observations about Performance

At 2 MBps

  1. None of the implementations could service 10K requests in a second on a Raspberry Pi 3B.
  2. Actix-Rust and Go implementations consistently performed ~2x better than all other (slower) implementations with Actix-Rust implementation performing better than Go implementation.
  3. Of the slower implementations, in all but one configuration, Trot-Elixir implementation consistently performed best. It was followed by Cowboy-Erlang, Flash+uWSGI, Phoenix-Elixir, and Vertx-Kotlin implementations in different concurrent requests configurations.
Performance at 2MBps [Click to enlarge]

Actix-Rust and Go implementations consistently performed ~2x better than the other considered web service implementations.

At 6 MBps

  1. Go implementation performed better than the Actix-Rust implementation in few concurrent requests configurations.
  2. NodeJS-JavaScript replaced Phoenix-Elixir in the set of implementations that immediately followed Trot-Elixir in different concurrent request configurations.
Performance at 6 MBps [Click to enlarge]

At 10 MBps

Performance at 10 MBps [Click to enlarge]

Why does the number of requests per second increase with number of concurrent requests?

None of the web service technologies could service 10K requests per second on a Raspberry Pi 3B.

Observations about Failures/Reliability

Based on Failing Clients

Least (Best-Case) / Most (Worst-Case) number of failed clients in an Ansible script execution in various configurations. [Click to enlarge]
  1. Across the board, Ktor-Kotlin, Vertx-Kotlin, and Flask+uWSGI-Python3 implementations were most reliable with zero failed clients in every execution.
  2. Actix-Rust, Go-Server, and Trot-Elixir implementations were the next most reliable implementations with no more than two failed clients in at most two configurations in when considering worst-case executions.
  3. In many configurations, NodeJS-Express-JavaScript, NodeJS-JavaScript, Micronaut-Kotlin, Ratpack-Kotlin, Phoenix-Elixir, Tornado-Python3, and Cowboy-Erlang implementations involved failed clients when considering worst-case executions.
  4. In some configurations, every execution of Micronaut-Kotlin, Tornado-Python3, and Cowboy-Erlang implementations involved at least one failed client. [The cells with red text.]

Based on Completed Requests

Number of successfully completed requests (X) and number of completed requests (Y) (in X/Y format) in various configurations when considering Ansible script executions with most number of completed requests (Best-Case). [Click to enlarge]
  1. While Flask+uWSGI-Python3 implementation did not involve failed clients, it involved failed requests in higher concurrent requests configurations.
  2. While the Phoenix-Elixir, Trot-Elixir, and Cowboy-Erlang implementations fared well in terms of the number of failed clients, they did not fare well in terms of the number of successfully completed requests.
Number of successfully completed requests (X) and number of completed requests (Y) (in X/Y format) in various configurations when considering Ansible script executions with least number of completed requests (Worst-Case). [Click to enlarge]
  1. Even the worst-case behaviour of Ktor-Kotlin and Vertx-Kotlin implementations did not involve failed requests!!
  2. If both performance and reliability were considered, then Vertx-Kotlin would be preferred over Ktor-Kotlin.
  3. Actix-Rust and Go implementations exhibited the next best worst-case behaviour (in order) with at most two executions involving failed requests.
  4. The implementations in the bottom-half involved failed requests in 1000, 1500, and 2500 concurrent requests configurations.
  5. Deviating from the best-case executions, NodeJS-*, Micronaut-Kotlin, Ratpack-Kotlin, and the implementations in the bottom half of the table exhibited failed requests in pretty much all configurations involving 1000 or more concurrent requests in all network traffic configurations.
  6. Ratpack-Kotlin, *-Elixir, Flash+uWSGI-Python3, and Cowboy-Erlang implementations involved failed requests in 500 concurrent requests configurations as well.
  7. More than 5% of the issued requests failed in almost all instances/configurations involving failed requests. [Highlighted in red]
  8. The data in the table suggests Cowboy-Erlang implementation failed at 100 concurrent requests and 6 MBps configuration. Upon closer examination of the raw data, no data was recorded for the executions of Apache Bench on three client nodes and the other two executions crashed prematurely. Further, the remaining four Ansible script execution completed successfully without failures. So, this 0/0 entry should be ignored.

Even the worst-case behaviour of Ktor-Kotlin and Vertx-Kotlin implementations did not involve any failed requests!!

Summary

  1. None of the web service technologies could serve 10K requests per second on a Raspberry Pi 3B.
  2. Actix-Rust and Go implementations were most performant.
  3. Ktor-Kotlin and Vertx-Kotlin were the most reliable implementations followed by Actix-Rust and Go implementations.
  4. If both performance and reliability were considered, then Vertx-Kotlin should be preferred over Ktor-Kotlin.

A Thought about Choosing Technologies

Source Code

Next Up

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Venkatesh-Prasad Ranganath

Software Engineer / Researcher interested in software engineering, programming languages/tools, systems, and data.