Save 60% on server costs with this setup

Use Varnish to speed up your high traffic website and to save more than 60% on your server

Eni Sinanaj
Nov 18, 2019 · 6 min read
HTTP requests on the web server that pass through Cloudflare

Context

Some friends of mine have created a network of websites on different topics which are visited by 5 million people monthly on average.

Everything was working fine but the server setup they had resulted in 2 issues. One was that it was not scalable. And the second issue was a bit more serious because the server provider decided out of the blue to close the account because of an allegedly terms infringement by the websites.

What we understood later is that the fixed payment (of 30€/year) was not covering the server costs at Aruba that had to work continuously for 4–5M hits a month.

After this I offered to find a suitable substitute for the server and moved it twice to some shared VPS instances and in the end decided to go with AWS or Azure. Because of issues with AWS to set up a EC2 instance near the region of my interest I chose Azure in the end.

To have the websites up and running as soon as possible I created an old-fashioned web server with Ubuntu, added certbot, php, nginx and what not and created the virtual hosts. At the same time I migrated the Wordpress instances to this new machine.

I put all the databases to an expensive Azure MySQL instance just to be sure it could handle the requests.

First problem

It occurred already the first day the sites were running again. As soon as there were 800 people simultaneously on the website the server would start responding with server errors. HTTP Status Code 500

The issue wasn’t clear to me as the processing unit (Ubuntu VM) and the MySQL machine as well were using 2% of CPU’s time. Very little to not be able to serve the requests.

With some help from some friends of mine, I found out I should have increased the number of FastCGI workers. After doing that, the server was able to serve more requests at the same time.

php-fpm warming up

Second problem

Number of requests being handled by the server was not as much as expected. The server was able to handle about 1.2 thousand requests at the same time. Not sufficient for us.

First thing I did was looking for the bottleneck. Usage was fine in the database instance but it was reaching 100% on the web server. So after getting this confirmation on the Azure console, I logged into the virtual machine and executed thetop command.

The result was pretty disturbing as I had 8 cores of the processor at 100% while stresstesting the server. And 20% of the requests where failing with a 500 server error response.

Solution

Since the issue was clear, it was due to too much processing needed by WordPress to create the pages, I though of adding a different number of cache layers.

Application level caching

First cache I installed was on application level. I used the Autoptimize plugin for WordPress. It was giving a good result but I needed more. Not just that but for some yet unknown reason, it messed up with Elementor (page builder for WordPress) and our front-end developers started complaining.

However I would suggest using Autoptimize if the tools permit it because it helps a lot with speeding up your website. I used it on another WordPress installation and it helped getting an A- rating from the previous F.

Web server caching

Next was enabling caching on web server level (NGinX in my case).
For that I added in the beginning of my server file in sites-enabled:

fastcgi_cache_path /etc/nginx/cache levels=1:2 keys_zone=ONLYO:500m inactive=30m;fastcgi_cache_key “$scheme$request_method$host$request_uri”;fastcgi_cache_use_stale error timeout invalid_header http_500;fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

Next part was inside the server block adding the following conditions to disable caching for POST requests

set $skip_cache 0;# POST requests and urls with a query string should always go to PHPif ($request_method = POST) {   set $skip_cache 1;}if ($query_string != "") {   set $skip_cache 1;}# Don't cache uris containing the following segmentsif ($request_uri ~* "/wp-admin/|/wp-json/|/xmlrpc.php|wp-.*.php|/feed/|index.php|sitemap(_index)?.xml") {   set $skip_cache 1;}# Don't use the cache for logged in users or recent commentersif ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_no_cache|wordpress_logged_in") {   set $skip_cache 1;}

Once the $skip_cache variable has been defined, we need to use it, again in the server block under location ~ \.php$.

location ~ \.php$ {   include snippets/fastcgi-php.conf;   fastcgi_pass unix:/run/php/php7.2-fpm.sock;   fastcgi_buffer_size 128k;   fastcgi_buffers 256 4k;   fastcgi_busy_buffers_size 256k;   fastcgi_temp_file_write_size 256k;   fastcgi_cache_bypass $skip_cache;   fastcgi_no_cache $skip_cache;   fastcgi_cache ONLYO;   fastcgi_cache_valid 512 30m;}

With the above configuration, the NGinX cache is enabled for most of the static requests to our webserver and get refreshed every 30 minutes, which is a reasonable duration for our specific case.

I also didn’t need my cache to be persistent since it’s being retained for only 30 minutes so I mapped the cache folder to the RAM directly.

Follow these instructions if you want to do the same: https://askubuntu.com/questions/597268/virtual-ram-folder-in-ubuntu

Note that you might want to make this configuration persistent so that after a server reboot you still would have the RAM mapped folder.

After setting up this cache the site started responding very quickly and the number of concurrent “users” that could be served where doubled.

Full page caching (with a reverse proxy)

Although my brother (Ani Sinanaj) suggested me to use something even more performant. The answer was Varnish. After looking a bit into it and understanding how it worked I started a new VM on Azure and installed Varnish on it.

Soon enough I figured out and fixed all the routes from the new varnish server to the backend NGinX web server. Just normal challenges that you always have with a reverse proxy installation such as forwarding protocol and port and what not for HTTPS to work properly.

After the website was responding I could start playing around with varnish to enable the caching.

The ultimate set up is to keep for 24 hours every static page/resource for every GET request regardless of the query string parameters. The website became lightning fast and all my virtual machines (web server, varnish server, database) were now working at 2% of their capacity.

At this point our monthly bill from azure was around 630€.

Before downgrading the VMs we paid a little less than the forecasted cost

I started downgrading all the machines for as long as they were only working at a reasonable percentage of their capacity.

Our webserver now is working at 5%, the database server consumes at most 4% while the cache server is working on average at 30% with peaks that reach 80%.

Cost forecast for the whole account now is estimated at about 70€ for November but I suppose the final bill will be between 120€ and 140€.

There are a couple of drawbacks with this configuration.

  1. Updating WordPress articles will not take effect immediately. More on this coming soon.
  2. With the downgraded hardware, the minute we stop the reverse proxy WM (varnish cache server) everything fails as soon as we reach 300 people online at the same time. From the database, unable to serve all the infinite requests to the web server itself.

Thanks to Ani Sinanaj for the hints on NGinX cache and Varnish

The Startup

Medium's largest active publication, followed by +563K people. Follow to join our community.

Thanks to Ani Sinanaj

Eni Sinanaj

Written by

#tech #startup #entrepreneur #business #money #excess #earth #motivational #speaker #hype #manager #startup #consulting

The Startup

Medium's largest active publication, followed by +563K people. Follow to join our community.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade