DHCP and resolv.conf options (part1 — Centos)

I’ve got a very simple task: add to our cloud images (Open Stack) a small feature — rotate DNS resolvers.

At principle solution is very simple — ‘add “options rotate” to /etc/resolv.conf’. But our /etc/resolv.conf is updated by DHCP client. How can I do this? Before I started to write this article I’ve tried all simple things: hooks for dhcpclient, manually updating resolv.conf, adding ‘manage-resolv-conf’ in cloud.cfg (for cloud-init), etc. I’ve done everything you normally do in this situation. And I come to the point I understand I need to write down all those bizarre things happen around combination of dhcpclient and resolv.conf.

(this is not a finished story and more like a travel log)

Centos 7

If I do

echo ‘options rotate’ >> /etc/resolv.conf && reboot

then, after boot, there are no dns-related dhcp settings in resolv.conf, only ‘options rotate’ line. I tried to remove the file and force dhcp to refresh a lease, and normal settings appeared.

The same thing happens if I add that line to /etc/dhcp/dhclient.d/anyting.sh (chmod +x). It adds option to resolv.conf, at every reboot (one more line on each lease renewal), but it does not add actual nameserver line from DHCP lease).

I know about ‘PEERDNS=no’ option. It is not enabled (I’ve grepped it through all /etc/*), and, moreover, it does not solve issue with disappeared nameservers (I didn’t overwrite file, I’ve added one line).

… OK, may be this is some kind of ‘keep manual’ magic.

Test: create a normal resolv.conf manually.

/etc/resolv.conf:

options rotate
nameserver 192.168.8.8
nameserver 192.168.8.88

reboot, look at content of /etc/resolv.conf:

options rotate

WUT?

Second try, add something into to check what would be kept.

before reboot:

#header
options rotate timeout:42
options rotate timeout:43
#line 1
nameserver 8.8.8.8
#line 2
nameserver 8.8.4.4
#keep this, please
#nameserver in comments
#options in comments

after reboot

options rotate timeout:42
options rotate timeout:43

They’re mocking me. Illuminati in my Linux are mocking me. Reptiloids?

The Documentation

Model:

dhclient called upon interface initialization by scripts in /etc/sysconfig/network-scripts/* (which one? ifup-eth? network-functions?) due to BOOTPROTO=”dhcp”.

dhclient-script is performing query, calling entry hooks and generating resolv.conf with a make_resolv_conf() function (man dhclient-script).

grep -r ‘make_resolv_conf’ /etc/*

no results.

… Checking a bash part… There is no bash part:

repoquery -l dhclient
/etc/NetworkManager
/etc/NetworkManager/dispatcher.d
/etc/NetworkManager/dispatcher.d/11-dhclient
/etc/dhcp
/etc/dhcp/dhclient.d
/usr/lib64/pm-utils/sleep.d/56dhclient
/usr/sbin/dhclient
/usr/sbin/dhclient-script
/usr/share/doc/dhclient-4.2.5
/usr/share/doc/dhclient-4.2.5/README.dhclient.d
/usr/share/doc/dhclient-4.2.5/dhclient.conf.example
/usr/share/doc/dhclient-4.2.5/dhclient6.conf.example
/usr/share/man/man5/dhclient.conf.5.gz
/usr/share/man/man5/dhclient.leases.5.gz
/usr/share/man/man8/dhclient-script.8.gz
/usr/share/man/man8/dhclient.8.gz
/var/lib/dhclient

Additionally, I saw entry in a journal and got proof it is done by dhclient-script, not by dhclient itself:

Mar 21 10:08:51 sdfs NET[751]: /usr/sbin/dhclient-script : updated /etc/resolv.conf

It’s hard to explain my happiness in the face of debugging of 805-liner in bash…

Suddenly, I saw lines:

 if [ -n “${RES_OPTIONS}” ]; then
     echo “options ${RES_OPTIONS}” >> ${rscf}
 fi

… RES_OPTIONS is never defined in the script. My guesses:

1. This is a bug
2. They want RES_OPTIONS somewhere from network-scripts.

I will test it with RES_OPTIONS in ifcfg-eth0. (reboot later) YES, it works.

I found how to fix this problem in Centos. Perfect. Now I need to do the same with Debian/Ubuntu (see part 2).

Meanwhile I’m struggling to understand causes of strangeness with original resolv.conf.

I’ve glanced upon script and I found it has many variables not defined anywhere in the script, like ${new_domain_name_servers}” ,“${old_domain_name_servers}”, etc. I suspected they are provided by dhclient.

For pure curiosity I want to know who and how called dhclient-script.

I’ve added a simple snippet to the script:

date >>/var/log/dh_log
env >>/var/log/dh_log
echo “cmd: $*” >>/var/log/dh_log
ps faux >>/var/log/dh_log

And I found /var/log/dh_log do not exist after reboot. It was never executed (or script have not enough permissions). I moved output to /tmp/dh_log.

Yep, there were not enough permissions. /tmp/dh_log has been created.

And, meanwhile, I found that previous bug has been reproduced regardless of RES_OPTIONS: if ‘options’ line is in the /etc/resolv.conf, on the next boot it replace new /etc/resolv.conf with only line ‘options …’.

F..k.

But I can see who and how running the script:

/sbin/dhclient -H sdfs -q -lf /var/lib/dhclient/dhclient...
  \_ /bin/bash /usr/sbin/dhclient-script
      \_ ps faux

And I can see environment variables:

requested_host_name=1
new_host_name=host-xxx
new_subnet_mask=255.255.255.224
new_domain_name=.com
requested_time_offset=1
requested_classless_static_routes=1
new_next_server=188.xx.xx.
new_ip_address=188.xx.xx.xx
new_network_number=188.xx.xx.xx
interface=eth0
requested_ntp_servers=1
reason=REBOOT
requested_domain_search=1
new_expiry=1459773667
PATH=/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
new_dhcp_lease_time=1209600
pid=665
new_dhcp_server_identifier=188.xx.xx.xx
PWD=/etc/sysconfig/network-scripts
new_dhcp_renewal_time=604800
requested_subnet_mask=1
new_routers=188.42.195.65
requested_nis_domain=1
requested_domain_name=1
new_domain_name_servers=192.168.8.8 192.168.8.88
SHLVL=1
requested_domain_name_servers=1
new_dhcp_message_type=5
new_dhcp_rebinding_time=1058400
requested_broadcast_address=1
new_broadcast_address=188.xx.xx.xx
requested_routers=1
requested_interface_mtu=1
requested_nis_servers=1
_=/bin/env

Interestingly, I can’t see any traces of ‘rotate’ or ‘options’. That means they are written not by dhclient-script, but something else (hooks or dhclient itself).

Before I continue, I have a hypothesis:

When dhclient doing ‘remove’ it would remove all information from lease from resolv.conf, except ‘options’. But it will keep resolv.conf unchanged if something (e.g. ‘options’ string) is in the file at the moment of the boot.

I done system upgrade, and it suddenly fixed. That means it was a bug and was fixed.

Anyway, answer for Centos7:

update, add RES_OPTIONS=”rotate” in /etc/sysconfig/network-scripts/ifcfg-eth0 (and eth1, etc).