Virtualized vs Bare-Metal Cloud Hosting
Update September 1st, 2017:
DigitalOcean released new High CPU Droplets so I have updated this post to include their benchmarks and have made small modifications to sections that have been affected.
Update October 24th, 2017:
Packet has an updated Type 1 Server (Kabylake) and a Type 2A server with an incredible 96-core ARM CPU, so I have updated this post to include their benchmarks and have made small modifications to sections that have been affected.
I’ve been a fan of DigitalOcean since they launched over five years ago and a very happy customer for over three years. They sold me on their pricing, interface and SSDs. Combined with Cloud66 for the low-level management, it’s an incredible solution. Fast, inexpensive, and flexible.
As our company and the demands on our infrastructure has grown, I’ve been curious to see what benefits there would be to switching to a bare-metal solution instead of a virtualized solution and what the current options look like.
That led to me to discover Packet, which looked to be very similar to DigitalOcean in terms of interface, spin-up times and cost but offered exclusively bare-metal servers instead of virtualized ones.
I wanted to see how the two compared with some simple benchmarks to see if it was worth switching to bare-metal. I came to a few interesting findings:
The $80 per month DigitalOcean Droplet was less performant than the $40 per month Droplet, by nearly half.
Use DigitalOcean up until the $40 per month Droplets then use Packet for any servers larger than that.
The Type 1 bare-metal server from Packet is an absolute powerhouse and fantastic value for a medium-sized server.
Update September 1st, 2017:
In general, the High CPU variants of DigitalOcean are nearly double the performance of their regular counterparts, sometimes more.
With the release of DigitalOcean’s High CPU Droplets, the gap has certainly closed between them and Packet.
The higher-end DigitalOcean High CPU Droplets provide excellent performance and value.
Update October 24th, 2017:
Packet’s Type 2A 96-core ARM servers are beasts when it comes to multi-threaded CPU and MySQL tasks. If you have a process server, for example doing video transcoding, that takes advantage of multithreading, then this is a great value.
I know some of you just want to see the results so I’ll start with the graphs and “show all of my work” further down in the post if you want to dig deeper or scrutinize my methodology.
Results
Raw Benchmarking Data
I collected all of the results in a Google Sheet so you can view it if you want to dig deeper into the data.
Feel free to comment on the Google Sheet and let me know if you see any errors or formatting mistakes.
If you hover over the name of the droplets, you’ll see a note popup with the system information.
You can click on the Geekbench ID to go to the online viewer of the results.
Multiple Trials
I conducted three trials of all the tests and averaged the results, with the exception of the sysbench filesystem I/O test, which I only conducted a single trial for, simply because it took a long time to run. For the various Geekbench4 benchmarks, I ran them 3 times but only uploaded the first results to their online viewer.
Geekbench4 was just released and there’s a bug in their online viewer tool that is not allowing me to rename each result — I’m getting a generic Rails error screen — so the best way to tell which result is which is by looking at the RAM on the system information.
Below you will find out the methodology used to conduct these benchmarks and how to reproduce every single test, should you wish.
Server Setup
DigitalOcean Droplets
- Ubuntu 16.04.1 x64
- SFO 1 Datacenter
- New Droplets created on September 1st, 2016.
- Nothing installed except the libraries explicitly mentioned in this article.
DigitalOcean High CPU Droplets
- Ubuntu 16.04.3 x64
- SFO 2 Datacenter
- New Droplets created on September 1st, 2017. (That’s a coincidence it is exactly 1 year later.)
- Nothing installed except the libraries explicitly mentioned in this article.
Packet Servers
- Ubuntu 16.04 LTS
- Sunnyvale, CA Datacenter
- New servers created on September 1st, 2016 and October 24th, 2017
- Nothing installed except the libraries explicitly mentioned in this article.
Geekbench
Geekbench 4 came out the day before conducting these benchmarks so I was excited that we could use the latest version for these benchmarks.
Geekbench4 does not come with an Arm-based build so I, unfortunately, could not run Geekbench on the Packet Type 2A server, which runs an Arm processor with 96 cores. I was really looking forward to that so hopefully they end up releasing one, which they say they are going to do.
1. Download and install the latest Geekbench for Linux distro.
wget http://cdn.primatelabs.com/Geekbench-4.0.0-Linux.tar.gz
tar -zxvf Geekbench-4.0.0-Linux.tar.gz
cd build.pulse/dist/Geekbench-4.0.0-Linux2. Enter in Geekbench license in order to run 64-bit version.
./geekbench4 -r your-email your-key3. Get the system info for the Google Sheet.
./geekbench4 --sysinfo4. Run the 64-Bit benchmark and automatically upload the results.
./geekbench4 --upload5. Run two subsequent 64-Bit benchmarks but do not upload them.
./geekbench4 --no-uploadSysbench
I generally followed along with this HowtoForge article. I modified the commands as necessary to fit my conditions.
Install sysbench
- Install sysbench.
apt-get --assume-yes install sysbench--assume-yes avoids the “continue?” prompt.
CPU
1. Run the CPU test for 1 thread and 100 threads.
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run2. Run the CPU test for 100 threads.
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=100 runFilesystem I/O
- Prepare for the benchmark by creating numerous files that add up to 10G.
sysbench --test=fileio --file-total-size=10G prepareThe choice of 10G is interesting. The idea is that you want to select a value that is higher than the available RAM in the server, otherwise the machine will just use the RAM and not have to go to disk. However, I also wanted to run the same test across the entire spectrum of servers for both DigitalOcean and Packet so I had to use a size that would fit in the smallest server in our tests. This was the 0.5GB Droplet, which has 20GB of SSD space, so I chose 10GB for the test. This might result in not actually testing the I/O system of the larger servers with more than 10GB of RAM, but I think the test is still relevant, especially when you compare the larger servers of DigitalOcean with that of Packet.
2. Run the benchmark with 1 thread.
sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 --num-threads=1 run3. Run the benchmark with 100 threads.
sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 --num-threads=100 run4. Cleanup the filesystem after the tests were complete.
sysbench --test=fileio --file-total-size=10G cleanupMySQL
- Install MySQL.
sudo apt-get update
sudo apt-get --assume-yes install mysql-serverWhen you install mysql-server, it’ll ask you for a password. Because this is a testing environment only and the servers will be decommissioned immediately after these benchmarks are complete, I decided not to use a password for the root user. Just press “enter” when it asks you. It really does not like you setting no password for the root user so it’ll ask you 2 more times. Just press enter for both of those times.
This next line was only necessary on the Packet servers to kick off the mysql service.
sudo service mysql startAll servers were confirmed to be running MySQL Version 14.14, Distrib 5.7.13 / 5.7.20, for Linux (x86_64).
2. Create “sbtest” database and “sbtest” user account.
mysql -u root
CREATE DATABASE sbtest;
CREATE USER sbtest@localhost;
GRANT ALL PRIVILEGES ON * . * TO sbtest@localhost;
FLUSH PRIVILEGES;
quit;3. Prepare sbtest database with 1,000,000 records.
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql prepare4. Run the benchmark with 1 thread.
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=1 run5. Run the benchmark with 100 threads.
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=100 run6. Cleanup the database after you’re finished.
sysbench --test=oltp --db-driver=mysql cleanupThanks
I want to give a quick thanks to Gal Shir for creating Chartico that I used for creating the benchmarking charts in the results section. It doesn’t do many things (like allow you to edit!) but what it does do, it does very well indeed.
Update September 1st, 2017:
Chartico was too limited to use with the addition of DigitalOcean’s High CPU benchmarks so I had to redo all my charts in Excel.
Without Bias
Finally, I’ll add this proactively in case it comes in the comments or in people’s minds: I am in no way affiliated with either company. This benchmarking was conducted entirely without bias and solely to help our company and anybody else looking to compare these two company’s servers.
