Migrate to Google Cloud DNS Using OctoDNS: Part 2

Introduction

In this multi-part series, we will install, configure, test and run OctoDNS to transfer on-prem DNS zones/records to Google Cloud DNS.

Part 1: Installing OctoDNS for Google Cloud DNS
Part 2: Migrate BIND Zone File to Google Cloud DNS using OctoDNS (This Post)

This is the 2nd post of the series. We are assuming that you have gone through Part 1 and you have successfully installed OctoDNS. We will be continuing from where we left off in Part 1.

In this post, we will configure OctoDNS to migrate BIND formatted zone file into Google Cloud DNS.

We are assuming that the readers have basic DNS, Linux administration and Google Cloud knowledge.

Configuring OctoDNS:

OctoDNS uses a config.yaml file. In this section, we will create the necessary configuration files.

Below is the config.yaml file that we will be using, comments are added to provide an explanation for each field:

---
# A list of supported providers can be found here
providers:
  config:
    # Each provider has a different class
    class: octodns.source.axfr.ZoneFileSource
    # The directory holding the zone files
    # Filenames should match zone name (eg. example.com.)
    directory: ./zones
    # Should sanity checks of the origin node be done
    # (optional, default true)
    check_origin: false
  google_cloud:
    # You can read more about the GoogleCloudProvider class here
    class: octodns.provider.googlecloud.GoogleCloudProvider
    # Optional: The project in which the Cloud DNS zones will be hosted
    project: paarthm-****-playground
    # Credentials file for a service_account or other account can be
    # specified with the GOOGLE_APPLICATION_CREDENTIALS environment
    # variable. (https://console.cloud.google.com/apis/credentials)
    # The File with the google credentials (not required). If used, the
    # "project" parameter needs to be set, else it will fall back to
    # the "default credentials"
    credentials_file: ./paarthm-****-playground-aec344393b5e.json
    # In this section we will be defining the source and the target for
    # the actual zone. As mentioned previously, the zone name should
    # exactly match the file name in the zones directory.
zones:
  paarthmahajan.com.:
    sources:
      - config
    targets:
      - google_cloud

1. Let’s create directories to host the config and the zone files:

(env) paarthm@octodns-test:~/octodns$ mkdir config zones
(env) paarthm@octodns-test:~/octodns$ ls
config env output.txt zones

2. Create the config file:

(env) paarthm@octodns-test:~/octodns$ cd ~/octodns/config
(env) paarthm@octodns-test:~/octodns/config$ vim config.yaml

You can edit the bold text and use the sample config provided below. Please note that in this example, we will not be using the credentials file and so, we have commented out the “credentials_file:” parameter in the config. As a result, OctoDNS will use the service account associated with the VM.

---
providers:
  config:
    class: octodns.source.axfr.ZoneFileSource
    # The directory holding the zone files
    # Filenames should match zone name (eg. example.com.)
    directory: ./zones
    # Should sanity checks of the origin node be done
    # (optional, default true)
    check_origin: false
  google_cloud:
    class: octodns.provider.googlecloud.GoogleCloudProvider
    project: paarthm-****-playground
    # credentials_file: ./paarthm-****-playground-aec344393b5e.json
zones:
  paarthmahajan.com.:
    sources:
      - config
    targets:
      - google_cloud

3. Let’s create the zone files

(env) paarthm@octodns-test:~/octodns$ cd ~/octodns/zones
(env) paarthm@octodns-test:~/octodns/zones$ vim paarthmahajan.com.

Below is the sample zone file that we will be using. Just to keep things simple, we only have a TXT, CNAME and an A record in the zone file:

paarthm@octodns-test:~/octodns$ cat zones/paarthmahajan.com.

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.3 <<>> paarthmahajan.com @localhost axfr
;; global options: +cmd
paarthmahajan.com. 3600 IN SOA  ns1.paarthmahajan.com admin.paarthmahajan.com. 1066495 3600 900 604800 60
paarthmahajan.com. 3600 IN NS   ns1.paarthmahajan.com
paarthmahajan.com. 3600 IN NS   ns2.paarthmahajan.com
txt.paarthmahajan.com. 300 IN TXT "I am about to go on Google Cloud DNS"
cname.paarthmahajan.com. 600 IN CNAME www.paarthmahajan.com.
www.paarthmahajan.com. 300 IN A 10.10.10.10

Test:

Now that all the configuration files are in place, let’s do a dry run to test the configuration.

(env) paarthm@octodns-test:~/octodns$ octodns-sync — config-file=./config/config.yaml

If your instance has proper permissions, the output should look something like this:

********************************************************************************
* paarthmahajan.com.
********************************************************************************
* google_cloud (GoogleCloudProvider)
*   Create Zone<paarthmahajan.com.>
*   Create <CnameRecord CNAME 600, cname.paarthmahajan.com., www.paarthmahajan.com.> (config)
*   Create <TxtRecord TXT 300, txt.paarthmahajan.com., ['I am about to go on Google Cloud DNS']> (config)
*   Create <ARecord A 300, www.paarthmahajan.com., ['10.10.10.10']> (config)
*   Summary: Creates=3, Updates=0, Deletes=0, Existing Records=0
********************************************************************************

Please note that this was just a dry run and nothing was pushed to Cloud DNS. As we can see from the output, OctoDNS will do the following:

• Create the zone as it does not exist.
• Create the TXT, CNAME and the A record.

Finally we have the summary section which lists all the operations which will be performed. In this case we are simply creating 3 records. We will see an update operation later in this post.

Do it:

So from the previous test, we can conclude that the configuration is working as expected. Let’s now push the zone to Cloud DNS. We will use the same command but with the ”doit” flag.

(env) paarthm@octodns-test:~/octodns$ octodns-sync — config-file=./config/config.yaml --doit

Now we should see some additional output after the summary section:

********************************************************************************
* paarthmahajan.com.
********************************************************************************
* google_cloud (GoogleCloudProvider)
*   Create Zone<paarthmahajan.com.>
*   Create <CnameRecord CNAME 600, cname.paarthmahajan.com., www.paarthmahajan.com.> (config)
*   Create <TxtRecord TXT 300, txt.paarthmahajan.com., ['I am about to go on Google Cloud DNS']> (config)
*   Create <ARecord A 300, www.paarthmahajan.com., ['10.10.10.10']> (config)
*   Summary: Creates=3, Updates=0, Deletes=0, Existing Records=0
********************************************************************************
2020-05-17T18:19:15  [140348441831168] INFO  GoogleCloudProvider[google_cloud] apply: making changes
2020-05-17T18:19:16  [140348441831168] INFO  GoogleCloudProvider[google_cloud] Created zone zone-paarthmahajan-com--ec0382da57c24b5eb5889d959e55a58a. Fqdn paarthmahajan.com..
2020-05-17T18:19:21  [140348441831168] INFO  Manager sync:   3 total changes

Once OctoDNS executes successfully, we will be able to see the zone in the Cloud DNS console. An important thing to notice here is that, by default, OctoDNS will create a public zone.

If you prefer to use private zones instead, then you need to create the zone beforehand. OctoDNS uses the DNS name to check if a zone exists or not.

paarthmahajan.com public zone created by OctoDNS

So, we deleted the public zone that we just created and now we will use a private zone.

paarthmahajan.com private zone created manually

NOTE: If you have a public and a private zone with the same DNS name, then OctoDNS will use the public zone.

Let’s do a dry run again:

(env) paarthm@octodns-test:~/octodns$ octodns-sync — config-file=./config/config.yaml

Do you see any difference in the output? Because the private zone exists already, OctoDNS is not trying to create a new zone.

********************************************************************************
* paarthmahajan.com.
********************************************************************************
* google_cloud (GoogleCloudProvider)
*   Create <CnameRecord CNAME 600, cname.paarthmahajan.com., www.paarthmahajan.com.> (config)
*   Create <TxtRecord TXT 300, txt.paarthmahajan.com., ['I am about to go on Google Cloud DNS']> (config)
*   Create <ARecord A 300, www.paarthmahajan.com., ['10.10.10.10']> (config)
*   Summary: Creates=3, Updates=0, Deletes=0, Existing Records=1
********************************************************************************

Run the command again with the “doit” flag to push the changes to Cloud DNS. As You can see from the screenshot below, all the records from our BIND zone file are now in Cloud DNS.

Updates:

So, using OctoDNS to transfer zones to Google Cloud was pretty easy. What if we want to update the zone we just updated? This is where OctoDNS beats the gcloud import command. If you remember, OctoDNS lets you manage DNS as code.

Update 1: Add a New A Record

Let’s add a new A record to the zone file.

(env) paarthm@octodns-test:~/octodns$ echo “arecord.paarthmahajan.com. 300 IN A 1.1.1.1” >> ./zones/paarthmahajan.com.

We can confirm that the zone file is updated.

(env) paarthm@octodns-test:~/octodns$ cat zones/paarthmahajan.com.
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.3 <<>> paarthmahajan.com @localhost axfr
;; global options: +cmd
paarthmahajan.com. 3600 IN SOA  ns1.paarthmahajan.com admin.paarthmahajan.com. 1066495 3600 900 604800 60
paarthmahajan.com. 3600 IN NS   ns1.paarthmahajan.com
paarthmahajan.com. 3600 IN NS   ns2.paarthmahajan.com
txt.paarthmahajan.com. 300 IN TXT       "I am about to go on Google Cloud DNS"
cname.paarthmahajan.com.        600 IN CNAME www.paarthmahajan.com.
www.paarthmahajan.com. 300 IN A 10.10.10.10
arecord.paarthmahajan.com. 300 IN A 1.1.1.1

Let’s do a dry run again:

(env) paarthm@octodns-test:~/octodns$ octodns-sync --config-file=./config/config.yaml

From the output we can see that we are only creating 1 new record and in total, there are 4 records.

********************************************************************************
* paarthmahajan.com.
********************************************************************************
* google_cloud (GoogleCloudProvider)
*   Create <ARecord A 300, arecord.paarthmahajan.com., ['1.1.1.1']> (config)
*   Summary: Creates=1, Updates=0, Deletes=0, Existing Records=4
********************************************************************************

Run the command again with the — doit flag to push the changes to Cloud DNS. Once done, check the zone in the Cloud DNS console. We can see that the new record is added to the existing zone.

Update 2: Delete and/or Update Records:

Now let’s delete the CNAME record and change the “www.paarthmahajan.com” to a multi-value record. Below is our new zone file, we can see that there is no CNAME record and the WWW record has 2 IPs, 10.10.10.10 and 11.11.11.11.

(env) paarthm@octodns-test:~/octodns$ cat zones/paarthmahajan.com.
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.el6_10.3 <<>> paarthmahajan.com @localhost axfr
;; global options: +cmd
paarthmahajan.com. 3600 IN SOA  ns1.paarthmahajan.com admin.paarthmahajan.com. 1066495 3600 900 604800 60
paarthmahajan.com. 3600 IN NS   ns1.paarthmahajan.com
paarthmahajan.com. 3600 IN NS   ns2.paarthmahajan.com
txt.paarthmahajan.com. 300 IN TXT       "I am about to go on Google Cloud DNS"
www.paarthmahajan.com. 300 IN A 11.11.11.11
www.paarthmahajan.com. 300 IN A 10.10.10.10
arecord.paarthmahajan.com. 300 IN A 1.1.1.1

Let’s do a dry run again:

(env) paarthm@octodns-test:~/octodns$ octodns-sync --config-file=./config/config.yaml

From the output, we can see that OctoDNS is deleting 1 record and updating 1 record.

********************************************************************************
* paarthmahajan.com.
********************************************************************************
* google_cloud (GoogleCloudProvider)
*   Delete <CnameRecord CNAME 600, cname.paarthmahajan.com., www.paarthmahajan.com.>
*   Update
*     <ARecord A 300, www.paarthmahajan.com., ['10.10.10.10']> ->
*     <ARecord A 300, www.paarthmahajan.com., ['10.10.10.10', '11.11.11.11']> (config)
*   Summary: Creates=0, Updates=1, Deletes=1, Existing Records=5
********************************************************************************

Real World Test:

Up until now, we have tested OctoDNS with a small zone file with just 6 records at max. As we mentioned earlier, OctoDNS comes in handy when we are trying to migrate large zone files to Cloud DNS. Let’s try it out.

Here we have a zone file with 1600+ records in it. As you can see from the output, we have 1618 lines in the zone file.

(env) paarthm@octodns-test:~/octodns$ cat zones/paarthmahajan.com. | wc -l
Output: 
1618

Lets do a dry run to see how OctoDNS will process this zone file.

(env) paarthm@octodns-test:~/octodns$ octodns-sync --config-file=./config/config.yaml

The output of the dry run is too large to share in this post, but from the summary section, we can see that OctoDNS will create 1005 new records & delete 4 records(created using the test zone file earlier).

Summary: Creates=1005, Updates=0, Deletes=4, Existing Records=5

OctoDNS is smart enough to know that there are multi-value records in the zone file. As a result, it only created 1005 records instead of 1600+.

Summary:

We have successfully configured and tested OctoDNS for our Cloud DNS use case. We would highly recommend checking OctoDNS’s official Git page to learn more about OctoDNS.