Speeding up CI/CD on dpkg-based systems

if you install packages with apt, you can make it faster

The problem: you have tons of packages to install. You are fine to wait in production (actually, you aren’t, but you have no options), but for CI/CD with ephemeral instances total ‘crash safety’ of package installation is just an annoying slowdown.

I found two ways to speed things for CI/CD. Both are breaking ‘apt/dpkg’ promise to keep your system (mostly) alive in case of crash or reboot, but for things you’ve just created to run your playbook (or other code) and will destroy right after test, this promise is useless. If it crashes, drop it, try again.

The key slowdown is in sync/fsync operations to ensure proper state in filesystem between operations.

There are two ways to disable it.

dpkg-level

Add this to /etc/dpkg/dpkg.cfg:

force-unsafe-io

(you need a fresh version of dpkg, Bullseye/Focal both are fresh enough).

This allows dpkg to run a bit faster. In my tests large ansible playbook (which does tons of other things) got a modest acceleration of a few seconds.

Ansible task to do this:

- name: Disable fsyncs for dpkg
  become: true
  lineinfile:
    line: force-unsafe-io
    state: present
    path: /etc/dpkg/dpkg.cfg

system-wide level

The second option is nuclear. It either speed everything for real, or break it (f.e. ceph-osd mkfs does not like this option, so, in this case it breaks). Of course, it breaks not only apt, but every database you have on your server.

We use eatmydata package to install a library to override ‘fsync’ calls with empty calls. Normally you call it like this:

eatmydata some_binary_to_fool

But you can do it in a nuclear fashion:

- name: Install libeatmydata
  become: true
  apt: name=libeatmydata1 state=present
- name: Disable fsync system-wide
  become: true
  lineinfile:
    line: /usr/lib/x86_64-linux-gnu/libeatmydata.so
    path: /etc/ld.so.preload
    state: present
    create: true

Warnings

First: those things are completely unacceptable on production servers, and they can be used only in environments, where instances (containers, etc) are killed as soon as something failed.

Second: if you are running on some fast hardware with hidden writeback (f.e. hardware raid with BBU in WriteBack mode, or some SSD with power loss protection), you won’t see much of a gain — someone is cheating already and you got no leftovers to steal.