Varnish resolves backend IP only at launch time (and how to fix it)

Imagine the following scenario: you have a docker swarm for container orchestration and in the Varnish backend the host is defined with the container name (could also be an IP, doesn’t really matter as long as it’s something static), for example:

backend default {
    .host = "nginx";
    ...
}

Now you run the deploy and all of a sudden Varnish shows 503 and nothing useful can be found in Varnish logs.

Turns out, Varnish caches the host IP internally once at a launch time, so if after the deployment, there were no changes to Varnish, it will internally resolve to the old “nginx” IP, which is no longer available because the container was re-created and new private IP was assigned.

Starting with Varnish version 7, it already includes the following extension, which can be used to avoid this problem -> https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc

To fix internal Varnish DNS resolution to resolve correctly the host, make the following adjustments to the Varnish .vcl config file:

backend default {
    .host = ""; # leave host empty here
    ...
}

# set up a dynamic director
# for more info, see https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc
sub vcl_init {
    new d = dynamic.director(
        port = "80", # port can be adjusted as needed
        ttl = 1s # ttl can be adjusted as needed
    );
}

sub vcl_recv {
    set req.backend_hint = d.backend("nginx"); # host goes here
    ....
}

Now your backend host IP will be resolved correctly also after deploys.

As noted, unfortunately, this is not added in Varnish versions below 7, so you will have to build and add that extension yourself.

Here is a link to Installation instructions from the module authors https://github.com/nigoroll/libvmod-dynamic/tree/master#installation

Update — it’s possible that this also fixes the issues (if you are using Docker/Docker Swarm) and no dynamic director is needed. If that’s the case, please let me know.