Terraforming a Serverless MongoDB Replica Set with Split Horizon DNS on Azure and Cloudflare
Occasionally, some scenarios require non-orthodox deployments, whether it’s due to a temporary state of a system such as moving from an on-prem deployment to the cloud, or as convenient way of testing and validating a deployment with all of its moving parts.
Such is the case of the example in this article, which deploys a MongoDB, a database that needs no introduction, replica set on Azure, but, and this is the not-so-orthodox area of this deployment, exposes it to the external world in addition to being available on an internal private virtual network.
What’s non-orthodox about it? Well, usually a database is not exposed directly to the outside world, instead, there are one or more software layers, which also execute additional logic and just store and retrieve data.
The Problem
If you’ve worked with MongoDB replica sets before, you should be familiar with the replica set connection string which includes multiple hosts:
mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myReplThis is where things get complicated, because if we need the hosts to be available from both the internal network and the external world, we need to use hosts names that can be resolved from both the internal network and the external world, and they have to be the same host names, as MongoDB replica sets are only aware of the host names that were used when they were joined to the replica set, so two different names per host won’t work.
The Solution — Split Horizon DNS
To solve the problem, something called a split-horizon DNS needs to be used, and although the name suggests something out-of-the-ordinary, it actually means using two DNS zones, one private internal and one public external, with the same name, such as example.com.
By doing so, we’ll be able to register a host named node1.example.com once with a private IP in the private internal zone and once with a public IP in the public external zone, making the MongoDB connection string shown above valid for both environments.
The Example
The complete Terraform based example, which deploys three MongoDB nodes, exposes them to the outside world using an Azure Firewall, registers the internal IP addresses to an Azure Private DNS and registers the public IP addresses to a public DNS zone on Cloudflare, can be found in this GitHub repository:
For brevity reasons, I won’t cover the entire example here, but only the sections that make the replica set and DNS zones work as required, starting with the bash script that is executed by each node on startup to join the replica set:
You’ll notice that this is actually a template loaded by Terraform, which also populates the ${replica_set} and ${members} originating from the main entry point like so:
Once the script creates a node running on Azure Container Instances using the mongo-node module which is part of the example:
An internal private IP address is registered to the private DNS zone (as part of the module’s script):
For the public side of our deployment, an Azure Firewall and associated public addresses are created (one IP address per node) using the firewall module:
And SNAT and network rules on the above firewall, that will forward the traffic from the external IP addresses to the matching internal MongoDB nodes using the network-rule and nat-rule modules:
Lastly, public DNS records are created on Cloudflare using the IP addresses that were created with the firewall using the cloudflare-record module:
Deployment Testing
To test the deployment, a MongoDB connection string similar to the following will be required (ensure you set node names and replica set name to whatever values you used):
mongodb://mongo1.example.com:27017,mongo2.example.com:27017,mongo3.example.com:27017/?replicaSet=mongo-setAnd either use the mongo command line tool, or something like MongoDBCompass which is available here:
Conclusion
As exposing databases to the external world is always risky, even when using a firewall, it is recommended to also restrict the rules even further by setting source IP addresses which can access the nodes and use transport layer encryption to protect the information passing between the consumers and the MongoDB nodes.
