Websocket Proxys with Nginx and the Beauty of Nixos

Robin Raymond

We’ve recently ran into the problem where we wanted to connect the website frontend to Docker containers via a websocket in order to receive real time updates. In the day and age of Let’s Encrypt we, of course, also wanted to serve our frontend via SSL — so it was essential that the websocket connects via the wss protocol (even though none of the data transmitted needs to be protected).


This is the problem…

This is where the conundrum began. By the nature of things, we don’t know the IP address of the containers beforehand, so there is no way we can use static DNS to route the requests. On the other hand we can’t use self signed certificates in the containers either, since this would require the end user to accept the certificate — a very undesirable side effect. As an additional constraint, the content of the container is visible to our end users, so we cannot put private keys of authority signed certificates in there.

Why is this so damn hard, all we want to do is to establish a secure websocket connection.

…and this is the problem.

We came up with the following solution. We use Nginx instances to proxy the requests and support upgrading to the wss protocol. This solves the first problem, it allows establishing a secure connections to containers without valid certificates. The remaining issue is that we want to essentially connect to random IPs. To solve this, we use the pattern matching of Nginx to parse the IP and forward the request to the correct target.

location /ipv4-regex {
proxy_pass http://$1:port/your/forward/location;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}

Now the only remaining part is to deploy this in a reproducible fashion. This is where Nixos shines. Combined with Nixops, this is almost trivial to do. Here is our deployment script.

{  wsproxy = { pkgs, ... }: {
networking.firewall.allowedTCPPorts = [ 80 443 ];
services.nginx = {
enable = true;
recommendedGzipSettings = true;
recommendedOptimisation = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
virtualHosts."your.host" = {
addSSL = true;
enableACME = true;
locations = {
"~ \"^/([\\d]{1,3}\\.[\\d]{1,3}\\.[\\d]{1,3}\\.[\\d]{1,3})$\"" = {
proxyPass = "http://$1:port/your/location";
proxyWebsockets = true;
};
};
};
};
};
}

These fifteen lines completely describe the system. Let me explain what is going on here.

  • wsproxy is the name of the server we are defining.
  • networking.firewall.allowedTCPPorts opens the ports for HTTP and HTTPS in the the firewall (which by default is enabled on Nixos)
  • services.nginx.enable = true installs and enables an Nginx instance. The rest is the configuration of Nginx.
  • recommended... set a bunch of recommended settings. For example, recommendedTlsSettings translate to the following config
ssl_session_cache shared:SSL:42m;                               ssl_session_timeout 23m;                               ssl_ecdh_curve secp384r1;                               ssl_prefer_server_ciphers on;                               ssl_stapling on;
ssl_stapling_verify on;

If you are interested in the other settings, take a look at the defining nginx.nix file.

  • virtualHost."websocket.example.com" defines a new virtual host in the Nginx configuation.
  • addSSL = true; enableACME = true; adds HTTPS to this virtual host and automatically requests and installs a certificate from Let’s Encrypt (how cool is that?). There are also options like forceSSL if you want to redirect all traffic through HTTPS.
  • "~ \"^/([\\d]{1,3}\\.[\\d]{1,3}\\.[\\d]{1,3}\\.[\\d]{1,3})$\"" is a naive regular expression to parse IPv4 addresses. It is actually not the one we are using, but in order to keep it shorter I’ve included this slightly naive version. Note that certain symbols have to be escaped in the string (namely " and \ )
  • proxyPass defines where the connection is proxied to.
  • proxyWebsockets turns on websocket proxying.

There are many more possible settings, check out the Nixos options (just put “nginx” into the search field).


Now we only need to pair this with a deployment script for Nixops. Since we deploy to AWS ec2, this is rather easy. Check out what other possibilities there are!

Are you interested in what we are using this for? Checkout www.cryzen.com for more information. We love Linux and free software over there!

Robin Raymond

Written by

CTO of Cryzen LLC

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade