Magento, Varnish and GeoIP

GeoIP is pretty neat. It allows you to dynamically adjust some parts of your website based on a country, county, city or even a street a user viewing your website is coming from. But you already know that, right?

Some basic databases mapping an IP address to a country can be found for free on the web. For more accurate ones you’d have to pay but that’s only fair since I’d imagine collecting this data in not necessarily easy in the first place. GeoIP is nice and dandy and…

You can use it with Magento!

A very common use case is to redirect customers to their specific language version of the website. Another simple example would be to change default currency for customers coming from various countries, like showing USD price for USA customers, EUR for EU customers, GBP for UK and so on.

Many people use it (including me) and for a good reason. It’s quite easy to implement and as mentioned above basic GeoIP databases are available for free. We’re not going to cover GeoIP implementation in this article, though. It deserves it’s own post and maybe I’ll write about it one day.

So what’s the problem?

Well… Varnish is the problem. Varnish does not care about where your customers are coming from. It’s just passing a request to the web server, then stores the HTML in internal cache and flushes it out to the user. If requested URL was already cached, Varnish will return cache immediately without hitting the backend of the application. The really cool thing about Varnish, though, is that it’s very fast (usually).

Since Varnish doesn’t care about a customer location then every user will get the same version of the page once it was cached. Varnish supports dynamic content hole punching with ESI or AJAX so use it to your advantage. But it won’t help with our example.

Lets assume that our Magento store supports 2 currencies: GBP and EUR. The default currency is GBP but customers are allowed to change it at will. Thanks to our analytics tools we see that many or our customers are browsing from European Union countries and they generate a total of around 40% of total traffic. In order to make their experience a bit better we decided that it would be good to switch default currency to EUR for those customers automatically. To do so, after locating customer with GeoIp extension we store a cookie with a value of currency symbol based on the result.

For Varnish integration we use Nexcessnet_Turpentine extension.

We see that everything is working perfectly fine when Varnish is disabled. But as soon as we enable it things are not so juicy any more. The first customer to generate the cache wins the race. If a customer from France opened a particular page as a first person and therefore got prices in Euro another customer coming to that very same page will also see Euro prices. Even if she’s from UK!

On a live website many people browse through all the available pages. After some time of having Varnish enabled we see that around 40% or pages render prices in Euros disregarding customer origin. Defeated, we disable Varnish or GeoIP.

Why is it happening? Because Varnish (with Turpentine at Magento side) will ignore most of the cookies! (Normally Varnish would pass every request containing cookies to the webserver, resulting in no fast cached content delivery).

VCL to the rescue!

Varnish Configuration Language (VCL) is a special language designed for quite easy Varnish configuration even in runtime. Its official documentation is available here. In short,

The VCL language is a small domain-specific language designed to be used to define request handling and document caching policies for Varnish Cache. When a new configuration is loaded, the varnishd management process translates the VCL code to C and compiles it to a shared object which is then dynamically linked into the server process.

Turpentine extension comes with a predefined VCL templates. It’s offering a template specific for each Varnish version including versions 2, 3 and 4. For the most part those templates are filled with VCL code which is optimized for Magento. At some places there are, however, placeholders which are dynamically replaced with more VCL or C code.

Placeholders are basically an arbitrary string enclosed within {{ and }}, e.g. {{default_backend}}. Most of those placeholders are replaced with content that is configurable via System Configuration in Magento Admin panel. When you are finished with Turpentine configuration then in Cache Management page you can dynamically apply this config to Varnish. Turpentine will take the template, replace placeholders, optionally remove comments and unnecessary white spaces and upload it directly to varnishd process. Thanks to that you don’t have to restart Varnish every time you make changes to VCL file.

Lets see how we can update VCL to take into account cookie from our example above.

Step 1. Define what’s needs to be done

We want Varnish to create hash (equivalent of a FPC cache key) that will contain our cookie value. Cookie name is customer_geoip_currency. In our case cookie value can be either GBP or EUR.

If customer visits the website for the first time or doesn’t have this cookie in their request we should force Varnish to pass this request to the web server.

After browsing through Turpentine’s code we see that paths to VCL templates are hardcoded. So if we want to make changes to the VCL file we have to rewrite the class that’s responsible for fetching the template. We follow the first rule of Magento development here: never touch core or third party code. Never! (Unless you have a pretty darn good reason for that)

Step 2. Create custom module

For the sake of this example we’ll call it Ventus_Turpentine. It will contain config.xml, a Model folder which will contain one model class and a misc folder to store our VCL template. In this example we’re using Varnish version 3.0.7. Because of that we have to rewrite Nexcessnet/Turpentine/Model/Varnish/ Configurator/Version3 class.

We rewrite one protected method in that class:

// Ventus/Turpentine/Model/Varnish/Configurator/Version3.php

protected function _getVclTemplateFilename($baseFilename) {
    $extensionDir = Mage::getModuleDir('', 'Ventus_Turpentine');
    return sprintf('%s/misc/%s', $extensionDir, $baseFilename);
}

In config.xml we add required rewrite:

<config>
    <modules>
        <Ventus_Turpentine>
            <version>0.1.0</version>
        </Ventus_Turpentine>
    </modules>
    <global>
        <models>
            <turpentine>
                <rewrite>                    <varnish_configurator_version3>Ventus_Turpentine_Model_Varnish_Configurator_Version3</varnish_configurator_version3>
                </rewrite>
            </turpentine>
        </models>
    </global>
</config>

Last thing to be done before we head to VCL changes we have to copy 2 files into misc folder from Turpentine extension misc folder: uuid.c and version-3.vcl.

Step 3. Update VCL template

There are 2 VCL sections or rather functions that we need to update: vcl_hash() and vcl_recv().

The vcl_hash() function is responsible for creating the hash of requested page. The hash is an equivalent of a cache key in Magento terms. Varnish will store generated page under hash key.

Turpentine does a really good job out of the box. It makes sure that admin, customer and checkout sections are not being cached at all. Locate the vcl_hash function and add the following code right before the return(hash); statement:

if (req.http.Cookie ~ "customer_geoip_currency=") {
    hash_data(regsub(req.http.Cookie, "^.*?customer_geoip_currency=([^;]*);*.*$", "\1"));
}

This code snippet translates to: if HTTP request contains cookie “customer_geoip_currency” then extract cookie value using regular expressions (regsub() function) and add it to the hash data. Later on Varnish will compute hash key from all provided hash data.

Now Varnish will add be using our cookie for caching pages. This means that as long as customers have customer_geoip_currency cookie set they will see different versions of the page depending on the cookie value. This will also work if a new session will hit the server. In that case the frontend cookie doesn’t exist and Varnish will pass the request further down. Turpentine’s default VCL template enforces Varnish to never cache pages for customer without frontend cookie. Why? Because otherwise it would be impossible to maintain customer sessions!

Now we’re left with an edge case. What if a customer has frontend cookie but doesn’t have customer_geoip_currency cookie? It will happen for some period of time after deploying our changes to production.

In such case customers would not get customer_geoip_currency cookie until their frontend session cookie either gets expired or is removed. But we can prevent it from happening but enforcing Varnish to pipe down the request to the web server. We can do it via vcl_recv() function.

In the rewritten VCL template find vcl_recv() function and put the following code between frontend cookie handling and static cache check:

# customer_geoip_currency cookie handling
if (req.http.Cookie !~ "customer_geoip_currency=" && !req.http.X-Varnish-Esi-Method) {
    return (pipe);
}

# Existing code
if ({{force_cache_static}} &&
        req.url ~ ".*\.(?:{{static_extensions}})(?=\?|&|$)") {
    (...)
}

Added condition means that if customer_geoip_currency is not present in the HTTP request then Varnish will not look up for it in its internal cache. It will pass everything down to the web server and then cache the response.

TL;DR;

By making simple amendments to Varnish VCL file we are able to control how HTTP requests are handled. We were able add a custom cookie value to Varnish hash key and therefore allow different versions of the page for customers coming from different countries.

Varnish is pretty awesome and Nexcess did a very good job with Turpentine. It is a widely used solution. No wonder that Magento 2 natively supports Varnish as a full page cache type and recommends it for production environment (Magento2 FPC should be used only on development environments).

Resources:

• https://www.varnish-cache.org/docs/3.0/reference/vcl.html
• https://github.com/nexcess/magento-turpentine
• https://kly.no/varnish/regex.txt

P.S. There is a GeoIP mod for Varnish available, so you don’t necessarily have to use cookies. Still, you would want to add customer country to the hash key. If left it for you to decide whether that approach would work for you or not. The example shown in this article was just a showcase.

Feel free to share with your experience or ask any questions.