Which is Better Compute Instance — EC2 or GCE? (Part 1)
Benchmarking Performance and Cost Between AWS EC2 And GCP GCE
This blog post is complementary to episode 1 of symplcloud which you can watch right below.
Introduction
As most of you already know, there are three major providers in the cloud industry. Amazon Web Service, Microsoft Azure and Google Cloud Platform.
As of 2019, AWS is the dominant market leader with 61% market share, followed by Azure with 27% and GCP with 12%. Just by looking at the market share it seems AWS clearly is the winner here.
But at the same time you can see a downward trend in AWS’s market share over time. So what does this tell you? Should I choose AWS because it’s the market leader? or Azure or GCP because they are catching up really fast?
This is not an easy question to answer. And I don’t think there’s one ultimate answer for everyone. Depending on the type and size of your business, the answer can be different. Also, since cloud providers change their rates and release new products and features all the time, an answer for today does not necessarily mean it’s the answer for tomorrow.
But still, knowing what’s best for me today is still very important because you can use this information to make valid comparisons. So, enough about the intro and let’s dive right into the main topic.
Compute Instances
Compute Instances, also known as the Virtual Machines are the fundamental building blocks in all cloud infrastructure. It’s just called differently depending on the provider. For example, AWS branded their compute instances as EC2 (Elastic Compute Cloud). In GCP, it’s called GCE (Google Compute Engine). And in Azure it’s Virtual Machine.
Here’s a list of few instance types in AWS. As you can see, each instance type has a unique combination of virtual CPUs and memory. And as you can easily imagine, different instance types are priced differently. For example, t2.medium and t3.medium are both using 2 virtual CPUs and 4GB of memory but when it comes down to an hourly rate, t2.medium ($0.0464/hr) is around 11.5% more expensive than t3.medium ($0.0416/hr) in Northern Virginia region (us-east-1) which is the AWS’s primary region in US.
Also t2 and t3 instances are burstable which means when your usage exceeds a certain baseline, you have to choose either pay more than the base rate for above baseline performance or have your instance perform at or below the predefined baseline level. When your instance is performing under the baseline it collects CPU Credits every hour and when your instance is performing over the baseline, it starts to use the credit it has collected.
Once all the credit is used and the instance is still performing over the baseline, then AWS will charge you for the over the baseline usage at $0.05/hr flat rate per 1vCPU. You can find out more about it in their documentation. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html
Interestingly, in GCP you can adjust the virtual CPU and Memory combination. Since the goal of this post is to find out which compute instance is better, I am going to create instances in GCE that matches the spec of EC2 instances and run some tests on them. For starter, I am going to set the memory of my n1-standard-1 instance to 2GB instead of 3.75GB so that I can test it against t2.small instance in EC2
Preparation
You can create the instance from the web console as shown above but in order to make sure it’s reproducible in the future, I wrote a Terraform template (https://github.com/symplcloud/terraform-templates).
Terraform is a tool that enables you to write your infrastructure as code in HCL (HashiCorp Configuration Language). You can simply download the Terraform binary from https://www.terraform.io/ and update the path to run it. Once installed, you can run terraform init to initialize a project and terraform plan to preview the changes. And finally you can run terraform apply to actually create/update/delete resources.
Of course, you will have to make some local changes to terraform.tfvars file such as setting the region you want to use or in GCP’s case the project you want to associate the resource with. Then create the VMs using the command terraform apply. Creating VMs usually take 10 to 30 seconds.
Now, the instances are created and ready to be tested. Both instances are created in Northern Virginia region and GCE instance has 2GB of memory instead of 3.75GB.
Test Plan
So here are the 6 test categories we are going to run tests.
- cpu test
- memory test
- file i/o test
- instance creation time
- stress test
- monthly cost
All tests are conducted on instances in Northern Virginia region for both AWS (us-east-1) and GCP (us-east4). All hardward tests (CPU, Memory, File I/O) are done with sysbench and create time was measured by executing terraform apply command.
Lastly, stress test was conducted from loader.io by making HTTP GET requests to a custom Node.js application on the instance that resizes a large image into much smaller one. Whoever is interested in reproducing the result are welcome to clone this repository and play with it. https://github.com/symplcloud/image-resize-express-sharp
t2.small vs n1-standard-1
Findings
Simply put, n1-standard-1 showed superiority in every test category but the monthly cost. It showed 4.1% faster computation, 725% faster memory ops, 13.8% faster file i/o, 83% faster create time, 50% more concurrent clients for $7.86 (46.8%) more monthly cost.
But even in the monthly cost, if you take the fact that t2.small being the burstable instance into consideration, n1-standard-1 may win in that category as well in some cases.
Because as I mentioned above burstable instances can only perform above baseline while you pay for the higher-than-baseline usage either using the CPU credit the instance collected while performing under the baseline or getting billed. And the baseline for t2.small and t3.small instances are set to 20%.
Let’s say your t2.small instance is operating at constant 50% CPU usage for a month, it will first burn the 30 launch credit that t2.small instance comes with which translates to 100 minutes of vCPU usage over the baseline in this scenario.
If you want to know more about how CPU Credits works in AWS, please refer to this document CPU Credits and Baseline Performance for Burstable Performance Instances on AWS User Guide for more details.
Conclusion
As can be seen from above, n1-standard-1 instance outperformed t2.small instance but can be more expensive if you are expecting a low CPU usage with an occasional performance burst. However, if you are expecting a constantly high usage of CPU, n1-standard-1 can actually be cheaper than t2.small instance.
Hope you find this information useful and before I close I want to say that the results can be different if I run the same experiment in different region or in different time. Here are the list of variables and configuration I used for the tests.
- Instances were created in Northern Virginia region
- Ubuntu 18.04 LTS was used as base image
- The test was conducted in September 26, 2019 from Toronto, Canada.
- Memory of n1-standard-1 instance was set to 2GB to match t2.small
- CPU Test —
sysbench cpu --cpu-max-prime=20000 --threads=2 run - Memory Test —
sysbench memory --threads=2 run - File I/O Test —
sysbench fileio — file-total-size=4G — file-num=64 prepare - Instance Create Time —
terraform apply - Stress test tool — loader.io
- Test app runtime for concurrent traffic — Node.js v10.15.3
- Test app source code — https://github.com/symplcloud/image-resize-express-sharp
What’s Next?
Here are other match ups I plan to do in the following posts.
- t3 vs n1
- t3 vs n2
- m4 vs n1
- m4 vs n2
- m5 vs n1
- m5 vs n2
Of course, I can do other tests such as EC2 vs Azure VM based on your feedback, so please feel free to leave a comment here or write a post in our reddit community https://www.reddit.com/r/symplcloud/ on the match ups you want to see in this series. Also, it would be even nicer if you run your own tests and share the results with me. Thank you very much!
