DHCP and resolv.conf (part2 — Debian/Ubuntu)

Continuation of the story: (part1).

Same goal: add “options rotate” to /etc/resolv.conf when using DHCP with Debian and Ubuntu images (for Openstack).

For Ubuntu and Debian there is no way for dhclient to add resolver options. I’ve checked it: /usr/sbin/dhclient-script do not notice RES_OPTIONS anyhow. Whole implementation of make_resolv_conf() much simpler than Centos, and it have no knobs to turn.

Next idea: use resolvconf. I don’t like it, but there are no other ways (except to pin down /etc/resolv.conf or create a hook for dhclient-script with modified version of make_resolv_conf — but this is way too intrusive for a simple fix). Resolvconf have template we can mangle as much as we want — and it provides alternative version of make_resolv_conf() in dhcp-script’s hook.

This idea was my initial, but there is a catch: resovlconf -u will fail if /etc/resolv.conf is an actual file (not a simlink to ../run/resolvconf/resolv.conf).

And here the catch: We building images with slightly patched diskimage builder (link). And I have no idea why and when, but image have /etc/resolv.conf as a file (not a symlink) after boot. So any changes to resolvconf templates are futile.

This is a detective story: diskimage builder logs for building ubuntu is few megs in size and (my hunch says so) there going to be no any records about creating the /etc/resolv.conf file.

My initial approach was simple: add post-install.d/40-fix-resolvconf with following code (decorations omitted):

rm -f /etc/resolv.conf
ln -s ../run/resolvconf/resolv.conf /etc/resolv.conf

But it didn’t succeed. New instances have /etc/resolv.conf file, not symlink.

Next step is (rather ugly) to debug a code: I want to see ‘what is resolv.conf’ at the build end.

According to documentation, the last part is cleanup.d. It runs outside chroot, and something like 99-debug would be 99% executed after any other pieces (which can change resolv.conf).

I’m adding following simple debug snippet (do not forget to chmod +x it):

#!/bin/sh
echo ********** DEBUG *********
echo ********** DEBUG *********
echo ********** DEBUG *********
echo ********** DEBUG *********
set -x 
ls -la $TARGET/etc/

After build I’ve checked build logs:

…
-rw-r — r — 1 root root 97 Mar 21 14:48 resolv.conf
…

As you can see, it is. Regardless it has been deleted…

And I found few important entries in the log:

+ check_break after-cleanup bash
+ echo ‘’
+ egrep -e ‘(,|^)after-cleanup(,|$)’ -q
+ lsattr /tmp/image.KHaBKROH/mnt/etc/resolv.conf
+ grep ‘^….i’
lsattr: Operation not supported While reading flags on /tmp/image.KHaBKROH/mnt/etc/resolv.conf
+ sudo rm -f /tmp/image.KHaBKROH/mnt/etc/resolv.conf
+ ‘[‘ -L /tmp/image.KHaBKROH/mnt/etc/resolv.conf.ORIG ‘]’
+ ‘[‘ -f /tmp/image.KHaBKROH/mnt/etc/resolv.conf.ORIG ‘]’
+ sudo mv /tmp/image.KHaBKROH/mnt/etc/resolv.conf.ORIG /tmp/image.KHaBKROH/mnt/etc/resolv.conf
+ unmount_dir /tmp/image.KHaBKROH/mnt/tmp

Why, people, why?

Citation from source code:

function finalise_base () {
 TARGET_ROOT=$TMP_MOUNT_PATH run_d cleanup
 # If the file has been set immutable, we probably want to keep it
 if lsattr $TMP_MOUNT_PATH/etc/resolv.conf | grep ‘^….i’ >/dev/null ; then
 # We’re keeping the contents of resolv.conf set in the elements,
 # so remove the old saved file
 sudo rm -f $TMP_MOUNT_PATH/etc/resolv.conf.ORIG
 else
 # Remove the resolv.conf we created above
 sudo rm -f $TMP_MOUNT_PATH/etc/resolv.conf
 # Move the original back
 if [ -L $TMP_MOUNT_PATH/etc/resolv.conf.ORIG ] || [ -f $TMP_MOUNT_PATH/etc/resolv.conf.ORIG ] ; then
 sudo mv $TMP_MOUNT_PATH/etc/resolv.conf.ORIG $TMP_MOUNT_PATH/etc/resolv.conf
 fi
 fi
 # Cleanup /tmp in the guest, so there is less cruft left there
 unmount_dir $TMP_MOUNT_PATH/tmp
 find $TMP_MOUNT_PATH/tmp -maxdepth 1 -mindepth 1 | xargs sudo rm -rf — one-file-system
}

Mina kirai! Daikirai!

Basically, diskimage builder have it’s own opinion about resolv.conf, and do something strange regardless of any ‘finalize.d’ modules.

finalise_base called directly from bin/disk-image-create:

run_d_in_target finalise
finalise_base

Basically, I can not do anything with resolv.conf in diskimage builder. It will place own version afterwards. I can pin resolv.conf with immutable flag (and it would keep it from been changed by DIB), but I can’t pin file absence.

Therefore, I should look to the cloud-init. May be it can remove old resolv.conf and create symlink. I need to do this before DHCP stage, though. I’m not that keen in cloud-init’s crazy code …

class abstractclassmethod(classmethod):
    “””A backport for abc.abstractclassmethod from Python 3.”””
   __isabstractmethod__ = True
   def __init__(self, func):
       func.__isabstractmethod__ = True
       super(abstractclassmethod, self).__init__(func)

But I found in very sparse documentation (that’s why I dislike cloud-init) something about script_per_once.

I’ve added simple code to fix issue to the /var/lib/cloud/scripts/per-once, and it works… Somehow. Due to unknown magic ‘options rotate’ appears in /etc/resolv.conf only after second boot.

My main hypothesis: this scripts run only after DHCP initialization (when /etc/resolv.conf was updated by dhcp-script), and have any influence on result only on the next reboot. To fix this I created some a crazy monster:

#!/bin/sh
mkdir -p /run/resolvconf || true
mv /etc/resolv.conf /var/run/resolvconf/resolv.conf || true
ln -s ../run/resolvconf/resolv.conf /etc/resolv.conf
resolvconf -u || true

(meanwhile I’ve updated my image-tests to check if ‘options rotate’ is available on 1st and 2nd boot…).

It worked. Kinda. Empty ‘options rotate’ — which is obvious due to resovconf miscall.

My next attempt is even more straightforward:

#!/bin/sh
mkdir -p /run/resolvconf || true
mv /etc/resolv.conf /var/run/resolvconf/resolv.conf || true
ln -s ../run/resolvconf/resolv.conf /etc/resolv.conf
echo ‘options rotate’ >> /etc/resolv.conf

and it works. Yes, this is ugly, but it works. And this is production.

I feel ashamed, but for now there is no other way to fix it in reasonable time. I could send patch to diskimage builder to allow disablement of resolv.conf mangling at the final stage, but it would take forever to pass review…