FizzBoom benchmark (call for participation)

Paul Biggar
Sep 23 · 4 min read
Image for post
Image for post

I’ve been working recently on a benchmark, to try and see how to get the most performance from the Dark backend. I’ve reimplemented the core of Dark in various languages and web frameworks, notably in OCaml, F# and Rust.

As a reminder, Dark is a language and platform for building backend web services and APIs. Implementation-wise, it’s basically an interpreter hooked to a webserver and a DB. The language is a statically-typed functional, immutable language which is garbage collected.

Dark users can write arbitrary code that runs on our server, including making HTTP calls to slow and badly-behaving 3rd-party web servers. This means we need to efficiently support both computation and IO on the server. This benchmark is meant to measure that.

FizzBoom

FuzzBuzz is a well-known interview question, that may or may not have been appropriate in a bygone era, but that remains in existence today due to interviews that have not moved on. It asks you to list 100 numbers, and if they are divisible by 3 or 5 you print “fizz” or “buzz” respectively.

FizzBoom is the same benchmark, except instead of printing “fizzbuzz” when a number is divisible by both 3 and 5, you instead make a HTTP call to a local server which takes 1 second to respond. Here’s what this looks like in Dark:

let range = List::range 1 100
List.map range \i ->
if i % 15 == 0
then
HttpClient::get "http://localhost:8000/delay/1"
else if i % 3 == 0
then "Fizz"
else if i % 5 == 0
then "Buzz"
else toString i

Call for participation

Typically, someone builds benchmarks and then releases them to the wild. Invariably, they’ve overlooked something, and hordes of language advocates decry how it’s unfair. To avoid this unfairness, I’d like to create an opportunity for language advocates to improve the benchmarks before I release them. I’ll discuss the goals of this below, but feel free to jump right to the issues if you prefer.

The benchmark measures two numbers, calculated using wrk:

  • requests per second for HTTP calls to /fizzbuzz, which returns FuzzBuzz as JSON

Preliminary numbers (thanks Chris for reporting these) indicate F# is great (25k req/s) and Rust and OCaml are doing ok (16k req/s each), for FizzBuzz.

Benchmark name        req/s
---------------------------------------------------------
fsharp-giraffe: 24962.95
fsharp-giraffe-async: 19476.78
fsharp-suave-async: 1147.71
fsharp-suave-partial-async: Skipping broken benchmark
ocaml-httpaf: 14034.62
ocaml-httpaf-lwt: 14158.74
rust-hyper: 15985.69
rust-hyper-async: Skipping broken benchmark

However, it also shows that we’re not doing async right on any platform — FizzBoom languishes at 1 req/s on all platforms. Obviously, this is because the code I wrote doesn’t work, and is not actually a reflection on the languages and frameworks.

Benchmark name                Req/s
--------------------------------------------------------
fsharp-giraffe: 1.00
fsharp-giraffe-async: 8.99
fsharp-suave-async: 1.00
fsharp-suave-partial-async: Skipping broken benchmark
ocaml-httpaf: 0.10
ocaml-httpaf-lwt: 0.10
rust-hyper: Invalid fizzboom output
rust-hyper-async: Skipping broken benchmark

Improving your favorite language’s performance

If you’re worried about your language doing well in the benchmark, or are simply looking to help, there are a number of things you can do:

  • optimize your language’s web server: I may have used a poorly performing web server, have it in a poor configuration, or have hooked things up poorly

I’m also interested, but very cautiously, in improving the implementation of the interpreters. The prime directive for the interpreters is that they’re easy to modify and extend: Dark is a language going under much change, so rewriting the interpreters with a JIT, or assembly, or peephole optimizations, is not something I’m interested in. But small implementation changes that have big wins are valuable, whether they’re applicable to all languages or show off specific features of your favorite language. All the same, I’m going to be quite conservative on this, I don’t want to turn this into a game where we write non-idiomatic code to squeeze out a win that no-one would want to maintain.

If you’d like to help, I’ve filed the issues above, and written down some rules to keep the benchmark fair. I hope to release some benchmark results soon, once I’ve got the majority of them working.

You can sign up for Dark here, and check out our progress in our contributor Slack or by watching our GitHub repo. Follow Dark (or me) on Twitter, or follow the blog using RSS.

Darklang

Updates on and from Dark — darklang.com

Paul Biggar

Written by

Tech entrepreneur, software engineer. Founder of Dark: https://darklang.com. Founder of CircleCI. Lover of chocolate and pastries.

Darklang

Darklang

Updates on and from Dark — darklang.com

Paul Biggar

Written by

Tech entrepreneur, software engineer. Founder of Dark: https://darklang.com. Founder of CircleCI. Lover of chocolate and pastries.

Darklang

Darklang

Updates on and from Dark — darklang.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store