Image creation and testing with HashiCorp Packer and ServerSpec
Intro
At SumUp we use various technologies and tools depending on their purpose and trade-offs.
Recently we started a few infrastructure-heavy projects.
We had to solve the problem of creating images (virtual appliances) for QEMU and VirtualBox for multiple projects:
- one project creates guests using a chain of QEMU images
- one project creates guest using a chain of VirtualBox images
What this article is going to cover as “hands-on” tech
- HashiCorp Packer (v1.3.x)
- ServerSpec (v2.41.x)
Source code:
First, why Packer and what are the use-cases?
Best check out the “Why” and “Use Cases” at https://www.packer.io/intro/why.html and https://www.packer.io/intro/use-cases.html
What is ServerSpec and what are the use-cases?
ServerSpec in a sense is a library with its own command runner and “resources” provider SpecInfra, all based around RSpec, the Ruby BDD test suite framework.
Sample code from https://serverspec.org/
require 'spec_helper' describe package('httpd'), :if => os[:family] == 'redhat' do
it { should be_installed }
end
It’s not the only test library and runner around RSpec, but it’s one with good popularity and there are good amount of resources to get your goal done.
One of the competition is https://www.inspec.io/ by Chef. It almost reads the same and provides the same resources, since there’s not much room for metaphors and lingo when you’re talking about infrastructure.
Another sample code, but this time using InSpec (you can see more about the packageresource at https://www.inspec.io/docs/reference/resources/package/)
require 'spec_helper'describe package('httpd'), :if => os[:family] == 'redhat' do
it { should be_installed }
end
As you can see they’re *can* be exactly the same at times.
What the ordinary Packer build flow looks like
What the Packer build flow will look like with tests
Preparing the HashiCorp Packer Debian image to build:
The full file looks like this: (we’re reviewing below specific parts what they give us as functionality) (ref: https://github.com/syndbg/sumup-blog-hashicorp-packer-test/blob/master/packer-base-debian.json)
{
"variables": {
"debian_version": "9.6.0",
"boot_wait": "5s",
"cpus": "1",
"disk_size": "5000",
"headless": "false",
"memory": "512",
"mirror": "http://mirror.host.ag/debian/debian-cd/",
"iso_checksum_url": "http://mirror.host.ag/debian/debian-cd/9.6.0/amd64/iso-cd/SHA512SUMS",
"ssh_timeout": "30m",
"host_adapter": null,
"rdp_bind_address": "127.0.0.1",
"must_run_tests": "false",
"must_sleep_before_tests": "false"
},
"provisioners": [
{
"type": "shell",
"scripts": [
"./files/base-debian/00_guest_additions.sh"
],
"only": ["virtualbox-base-debian"]
},
{
"type": "shell",
"scripts": [
"./files/base-debian/00_qemu_agent.sh"
],
"only": ["qemu-base-debian"]
},
{
"type": "shell",
"scripts": [
"./files/base-debian/01_ssh.sh"
]
},
{
"type": "shell",
"scripts": [
"./files/base-debian/02_disable_lvmetad.sh"
]
},
{
"type": "shell",
"scripts": [
"./files/base-debian/03_logging.sh"
]
},
{
"type": "shell",
"scripts": [
"./files/cleanup.sh"
]
},
{
"type": "shell-local",
"script": "./serverspec/sleep_before_tests_packer.sh",
"environment_vars": [
"MUST_SLEEP_BEFORE_TESTS={{ user `must_sleep_before_tests` }}"
]
},
{
"type": "shell-local",
"command": "./serverspec/run_packer.sh ./spec/base_debian_spec.rb",
"environment_vars": [
"MUST_RUN_TESTS={{ user `must_run_tests` }}",
"SUDO_PASSWORD=packer123",
"TARGET_PASSWORD=packer123",
"TARGET_USER=root",
"BUILDER_TYPE={{ build_type }}"
]
}
],
"builders": [
{
"name": "virtualbox-base-debian",
"type": "virtualbox-iso",
"boot_command": [
"<esc><wait>",
"install <wait>",
"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg <wait>",
"DEBCONF_DEBUG=5 <wait>",
"debian-installer=en_US <wait>",
"auto=true <wait>",
"priority=high <wait>",
"locale=en_US <wait>",
"kbd-chooser/method=us <wait>",
"keyboard-configuration/xkb-keymap=us <wait>",
"netcfg/get_hostname=debian <wait>",
"netcfg/get_domain= <wait>",
"netcfg/choose_interface=auto <wait>",
"fb=false <wait>",
"debconf/frontend=noninteractive <wait>",
"console-setup/ask_detect=false <wait>",
"console-keymaps-at/keymap=us <wait>",
"<enter><wait>"
],
"boot_wait": "{{ user `boot_wait` }}",
"disk_size": "{{ user `disk_size` }}",
"format": "ova",
"guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
"guest_os_type": "Debian_64",
"headless": "{{ user `headless` }}",
"http_directory": "http",
"iso_checksum_type": "sha512",
"iso_checksum_url": "{{ user `iso_checksum_url` }}",
"iso_urls": [
"iso/debian-{{ user `debian_version` }}-amd64-netinst.iso",
"{{ user `mirror` }}/{{ user `debian_version` }}/amd64/iso-cd/debian-{{ user `debian_version` }}-amd64-netinst.iso"
],
"output_directory": "output-{{ build_name }}",
"shutdown_command": "systemctl poweroff",
"ssh_password": "packer123",
"ssh_timeout": "{{ user `ssh_timeout` }}",
"ssh_username": "root",
"virtualbox_version_file": ".vbox_version",
"vrdp_bind_address": "{{ user `rdp_bind_address` }}",
"vboxmanage": [
[
"modifyvm",
"{{ .Name }}",
"--memory",
"{{ user `memory` }}"
],
[
"modifyvm",
"{{ .Name }}",
"--cpus",
"{{ user `cpus` }}"
],
[
"modifyvm",
"{{ .Name }}",
"--nic1",
"nat",
"--nictype1",
"virtio"
],
[
"modifyvm",
"{{ .Name }}",
"--nic2",
"hostonly",
"--nictype2",
"virtio",
"--hostonlyadapter2",
"{{ user `host_adapter` }}"
]
],
"vm_name": "packer-{{ build_name }}"
},
{
"name": "qemu-base-debian",
"type": "qemu",
"accelerator": "kvm",
"boot_command": [
"<esc><wait>",
"install <wait>",
"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/qemu-preseed.cfg <wait>",
"DEBCONF_DEBUG=5 <wait>",
"debian-installer=en_US <wait>",
"auto=true <wait>",
"priority=high <wait>",
"locale=en_US <wait>",
"kbd-chooser/method=us <wait>",
"keyboard-configuration/xkb-keymap=us <wait>",
"netcfg/get_hostname=debian <wait>",
"netcfg/get_domain= <wait>",
"netcfg/choose_interface=auto <wait>",
"fb=false <wait>",
"debconf/frontend=noninteractive <wait>",
"console-setup/ask_detect=false <wait>",
"console-keymaps-at/keymap=us <wait>",
"<enter><wait>"
],
"boot_wait": "{{ user `boot_wait` }}",
"disk_cache": "writeback",
"disk_compression": true,
"disk_interface": "virtio",
"disk_size": "{{ user `disk_size` }}",
"format": "qcow2",
"headless": "{{ user `headless` }}",
"http_directory": "http",
"iso_checksum_type": "sha512",
"iso_checksum_url": "{{ user `iso_checksum_url` }}",
"iso_urls": [
"iso/debian-{{ user `debian_version` }}-amd64-netinst.iso",
"{{ user `mirror` }}/{{ user `debian_version` }}/amd64/iso-cd/debian-{{ user `debian_version` }}-amd64-netinst.iso"
],
"machine_type": "pc",
"net_device": "virtio-net-pci",
"output_directory": "output-{{ build_name }}",
"shutdown_command": "systemctl poweroff",
"ssh_password": "packer123",
"ssh_timeout": "{{ user `ssh_timeout` }}",
"ssh_username": "root",
"vnc_bind_address": "{{ user `rdp_bind_address` }}",
"qemuargs": [
[
"-m",
"{{ user `memory` }}"
],
[
"-smp",
"{{ user `cpus` }}"
]
],
"vm_name": "packer-{{ build_name }}"
}
]
}We use Debian 9.6 now, but we started with 9.5.
To be able to change version, mirror and checksum URL we expose the following user variables (ref: https://www.packer.io/docs/templates/user-variables.html).
"variables": {
"debian_version": "9.6.0",
"mirror": "http://mirror.host.ag/debian/debian-cd/",
"iso_checksum_url": "http://mirror.host.ag/debian/debian-cd/9.6.0/amd64/iso-cd/SHA512SUMS"
},And use them like this in the builders
"iso_checksum_type": "sha512",
"iso_checksum_url": "{{ user `iso_checksum_url` }}",
"iso_urls": [
"iso/debian-{{ user `debian_version` }}-amd64-netinst.iso",
"{{ user `mirror` }}/{{ user `debian_version` }}/amd64/iso-cd/debian-{{ user `debian_version` }}-amd64-netinst.iso"
],It’s important to note that the first path in iso_urls allows us to use a local iso and skip the download. Useful if run in network bandwidth constrained environment.
We run the following shell provisioners (more about them at https://www.packer.io/docs/provisioners/shell.html)
- virtualbox guest additions, if it’s a virtualbox image
- QEMU guest agent, if it’s a virtualbox image
- SSH authorized_keys configuration
- disabling of LVM metad service
- cleanup
The above provisioners are the ones that we must test.
Let’s first try running the packer build without tests in mind.
> packer build -var 'headless=true' \
-var 'host_adapter=vboxnet0' \
-only=virtualbox-base-debian \
-force packer-base-debian.jsonThe above will not run the test suite. It’ll provision from a Debian 9.6 ISO, unattended/automated installation of Debian with debian-installer preseed configuration.
We expect the following output
virtualbox-base-debian output will be in this color.==> virtualbox-base-debian: Cannot find "Default Guest Additions ISO" in vboxmanage output (or it is empty)
==> virtualbox-base-debian: Retrieving Guest additions checksums
1 items: 4.47 KiB / 4.47 KiB [==============================================================================================================================================================================================================================================] 0s
virtualbox-base-debian: Transferred: https://download.virtualbox.org/virtualbox/5.2.10/SHA256SUMS
==> virtualbox-base-debian: Retrieving Guest additions
virtualbox-base-debian: Found already downloaded, initial checksum matched, no download needed: https://download.virtualbox.org/virtualbox/5.2.10/VBoxGuestAdditions_5.2.10.iso
==> virtualbox-base-debian: Retrieving ISO
virtualbox-base-debian: Found already downloaded, initial checksum matched, no download needed: http://mirror.host.ag/debian/debian-cd//9.6.0/amd64/iso-cd/debian-9.6.0-amd64-netinst.iso
==> virtualbox-base-debian: Starting HTTP server on port 8361
==> virtualbox-base-debian: Creating virtual machine...
==> virtualbox-base-debian: Creating hard drive...
==> virtualbox-base-debian: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 3486)
==> virtualbox-base-debian: Executing custom VBoxManage commands...
virtualbox-base-debian: Executing: modifyvm packer-virtualbox-base-debian --memory 512
virtualbox-base-debian: Executing: modifyvm packer-virtualbox-base-debian --cpus 1
virtualbox-base-debian: Executing: modifyvm packer-virtualbox-base-debian --nic1 nat --nictype1 virtio
virtualbox-base-debian: Executing: modifyvm packer-virtualbox-base-debian --nic2 hostonly --nictype2 virtio --hostonlyadapter2 vboxnet0
==> virtualbox-base-debian: Starting the virtual machine...
virtualbox-base-debian: The VM will be run headless, without a GUI. If you want to
virtualbox-base-debian: view the screen of the VM, connect via VRDP without a password to
virtualbox-base-debian: rdp://127.0.0.1:5958
==> virtualbox-base-debian: Waiting 5s for boot...
==> virtualbox-base-debian: Typing the boot command...
==> virtualbox-base-debian: Connected to SSH!
==> virtualbox-base-debian: Uploading VirtualBox version info (5.2.10)
==> virtualbox-base-debian: Uploading VirtualBox guest additions ISO...
==> virtualbox-base-debian: Provisioning with shell script: ./files/base-debian/00_guest_additions.sh<omitted output for readability>
==> virtualbox-base-debian: Provisioning with shell script: ./files/base-debian/01_ssh.sh
<omitted output for readability>
==> virtualbox-base-debian: Provisioning with shell script: ./files/base-debian/02_disable_lvmetad.sh
==> virtualbox-base-debian: Provisioning with shell script: ./files/base-debian/03_logging.sh
<omitted output for readability>
==> virtualbox-base-debian: Provisioning with shell script: ./files/cleanup.sh
<omitted output for readability>
==> virtualbox-base-debian: Running local shell script: ./serverspec/sleep_before_tests_packer.sh
==> virtualbox-base-debian: Running local shell script: /tmp/packer-<omitted output for readability>
==> virtualbox-base-debian: Gracefully halting virtual machine...
==> virtualbox-base-debian: Preparing to export machine...
virtualbox-base-debian: Deleting forwarded port mapping for the communicator (SSH, WinRM, etc) (host port 3486)
==> virtualbox-base-debian: Exporting virtual machine...
virtualbox-base-debian: Executing: export packer-virtualbox-base-debian --output output-virtualbox-base-debian/packer-virtualbox-base-debian.ova
==> virtualbox-base-debian: Deregistering and deleting VM...
Build 'virtualbox-base-debian' finished.==> Builds finished. The artifacts of successful builds are:
--> virtualbox-base-debian: VM files in directory: output-virtualbox-base-debian
If you see very similar to the above output — it’s working and we can move on to the actual test suite.
How to run the test suite
There were really only two choices:
- Run the test suite on the guest
- Run the test suite on the host
“Run the test suite on the guest”:
- (+) No need to have runtime and dependencies on the host
- (=) Have “bootstrap” script for runtime and dependencies to be provisioned and executed to the guest
- (-) Have “cleanup” script to remove all runtime and dependencies and also the test suite output such as coverage reports
- (-) More complex for scenarios where you want to extract test artifacts such as coverage reports
- (-) Almost surely double-work in terms of cleanup
“Run the test suite on the host”:
- (-) Need to have runtime and dependencies on the host
- (+) No need to have “bootstrap” scripts for runtime and dependencies to be provisioned, executed and cleaned up from the guest
- (+) Test artifacts such as coverage reports will be already present on the host and easier to feed to other systems
- (+) No double-work and complexity of cleanup
Executing the strategy of choice — “Run the test suite on the host”:
We’re going to run the test suite via a shell-local provisioner (read more at https://www.packer.io/docs/provisioners/shell-local.html).
Since shell-local will run it from our host machine to the guest we need to solve two problems:
- what credentials we can use to execute shell commands on the guest to verify behaviour
- what’s the SSH IP and port we can use
The “1.” is solvable easily since we’re the ones creating and provisioning the image. For simplicity sake, we’ll use the root user — root and password packer123 .
About “2.”, though
If you’ve been using HashiCorp Packer for a while, you probably have read the chain of threads:
- https://github.com/hashicorp/packer/issues/3331
- https://github.com/hashicorp/packer/issues/3889
- https://github.com/hashicorp/packer/issues/4701 and more.
The verdict of that chain of threads — you can’t get your SSH communicator’s IP and port in the template.
Getting the IP and port of the guest created by Packer options
As the last choice regarding the testing strategy, this is also a binary choice.
Choice 1. Get the guest IP from the guest machine’s network interface
This can be done by executing a script with shell provisioner and retrieving the IP, saving the IP at for example /tmp/myip , downloading the file from the guest to the host using the file provisioner in download direction (read more at https://www.packer.io/docs/provisioners/file.html).
However, the above choice has a huge flaw — by default created guests have NAT-ed networking. You won’t be able to access them with their 10.x.x.x/8 CIDR IPs without extra hassle to port-forward SSH port 22.
Also if you don’t want to do the above, you can get away with using hostonly network adapters (as per VirtualBox terminology), but then you lose the ability to test NAT-only images.
Now let’s get to choice 2, which is the one we’ve picked.
Choice 2. Re-use Packer’s SSH port forwarding used for the SSH communicator
If we enable Packer to log with PACKER_LOG=1 and PACKER_LOG_PATH=packer.log we can parse the log file and the “verbose” output to get the following line
==> virtualbox-base-debian: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 3486)We can know how to get the port now. We can parse it since the port will be changing always. We can use a static port, but this is sometimes too naive luxury in dynamic environments such as CI.
The SSH address is known from a log message like:
==> virtualbox-base-debian: Using ssh communicator to connect: 127.0.0.1We won’t be parsing it since, so far based on Packer experience, it’s always going to be localhost.
This results in the following “glue” script used to make our test suite run against the guest (ref: https://github.com/syndbg/sumup-blog-hashicorp-packer-test/blob/master/serverspec/run_packer.sh)
#!/usr/bin/env bash
# NOTE: This file is run by packer provisioner `shell-local`.
# Tests on developer machines are run inside `serverspec` dir using `bundle exec rspec`.# Arguments:
# $1 = file path to the image-specific spec to run inside serverspec dir
set -exif [[ "$MUST_RUN_TESTS" != "true" ]]; then
exit 0
fiLOG="packer.log"ip_and_port="$(grep -F 'SSH handshake err: ssh: handshake failed: read tcp' "$LOG" \
| pcregrep -o1 '\->([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{1,5})' \
| sort \
| uniq \
| head)"if [[ "$ip_and_port" == "" ]]; then
port="$(grep -F 'Found port for communicator (SSH, WinRM, etc): ' "$LOG" \
| pcregrep -o1 'Found\s+port\s+for\s+communicator\s+.+:\s+([0-9]{1,5})' \
| sort \
| uniq \
| head)"if [[ "$port" == "" ]]; then
port="$(grep -F 'Creating forwarded port mapping for communicator (SSH, WinRM, etc) ' "$LOG" \
| pcregrep -o1 'Creating\s+forwarded\s+port\s+mapping.+\s+([0-9]{1,5})' \
| sort \
| uniq \
| head)"
fi
# NOTE: Packer uses localhost almost always.
# parse host from log message `Using ssh communicator to connect: ` only if needed.
ip_and_port="127.0.0.1:$port"
fiecho "Found IP and port in $LOG: $ip_and_port"export TARGET_HOST=$(echo $ip_and_port | cut -d ":" -f1)
export TARGET_PORT=$(echo $ip_and_port | cut -d ":" -f2)echo "Going to use host: $TARGET_HOST"
echo "Going to use port: $TARGET_PORT"
echo "Going to use user: $TARGET_USER"cd ./serverspec && bundle exec rspec $1
(more cases are handled since QEMU and Virtualbox builders don’t have consistent log messages in between them)
We integrate the above run_packer.sh glue code in the Packer debian template using the following provisioner
{
"type": "shell-local",
"command": "./serverspec/run_packer.sh ./spec/base_debian_spec.rb",
"environment_vars": [
"MUST_RUN_TESTS={{ user `must_run_tests` }}",
"SUDO_PASSWORD=packer123",
"TARGET_PASSWORD=packer123",
"TARGET_USER=root",
"BUILDER_TYPE={{ build_type }}"
]
}We specify whether to skip or run the test via Packer user variable must_run_tests specified like packer build -var must_run_tests=true|false and we explicitly give the BUILDER_TYPE via Packer template builtin variable build_type (read more about built-in template variables https://www.packer.io/docs/templates/engine.html).
The BUILDER_TYPE is going to be useful for us to test positively/negatively whether certain packages are present or missing if we use QEMU or VirtualBox based images.
The test suite structure:
serverspec
├── Gemfile
├── Gemfile.lock
├── run_packer.sh (the glue code)
├── sleep_before_tests_packer.sh (will be covered later)
└── spec
├── base_debian_spec.rb
├── goldenfiles
│ ├── logging
│ │ └── journald.conf
│ └── ssh
│ ├── authorized_keys
│ ├── root_rsa.pub
│ └── users_rsa.pub
├── spec_helper.rb
└── support
└── shared_examples
├── disable_lvmetad.rb
├── logging.rb
├── qemu_guest_agent.rb
├── ssh.rb
└── virtualbox_guest_additions.rbWe follow proven best practices from http://betterspecs.org/.
We use bundler to manage our dependencies.
RSpec + ServerSpec (resources and command runner) are used to verify state.
We also use test if provisioned files match using golden files (https://softwareengineering.stackexchange.com/a/358792) via the https://rubygems.org/gems/rspec-golden-files gem.
Let’s look at our spec_helper.rb
# frozen_string_literal: truerequire 'serverspec'
require 'net/ssh'
require 'rspec-golden-files'Dir[
File.expand_path(
File.join(
File.dirname(__FILE__),
'support', '**', '*.rb'
)
)
].each { |f| require f }set :backend, :ssh
set :sudo_password, ENV['SUDO_PASSWORD']
set :paranoid, falseoptions = Net::SSH::Config.for(ENV['TARGET_HOST'])options[:user] = ENV['TARGET_USER'] if ENV['TARGET_USER']
options[:user] ||= Etc.getlogin
options[:port] = ENV['TARGET_PORT'] if ENV['TARGET_PORT']
options[:password] = ENV['TARGET_PASSWORD'] if ENV['TARGET_PASSWORD']
options[:auth_methods] = %w[password publickey]set :host, options[:host_name] || ENV['TARGET_HOST']
set :ssh_options, options# NOTE: Disable sudo since it's not used.
# Tests are assumed to be run as root.
set :disable_sudo, trueif !ENV['BUILDER_TYPE'] || ENV['BUILDER_TYPE'].empty?
raise 'no environment variable `BUILDER_TYPE` provided'
endis_virtualbox = ENV['BUILDER_TYPE']&.start_with?('virtualbox')
is_qemu = ENV['BUILDER_TYPE'] == 'qemu'RSpec.configure do |config|
# NOTE: Disable `should` syntax.
# Expect is the only absolute.
# ref: http://www.betterspecs.org/#should
config.expect_with :rspec do |expectations|
expectations.syntax = :expect
endconfig.include RSpec::GoldenFiles
config.filter_run_excluding is_virtualbox: !is_virtualbox
config.filter_run_excluding is_not_virtualbox: is_virtualbox
config.filter_run_excluding is_qemu: !is_qemu
config.filter_run_excluding is_not_qemu: is_qemu
end# Set environment variables
# set :env, :LANG => 'C', :LC_MESSAGES => 'C'# NOTE: Set PATH since
We provide accurate SSH communication options via
options = Net::SSH::Config.for(ENV['TARGET_HOST'])options[:user] = ENV['TARGET_USER'] if ENV['TARGET_USER']
options[:user] ||= Etc.getlogin
options[:port] = ENV['TARGET_PORT'] if ENV['TARGET_PORT']
options[:password] = ENV['TARGET_PASSWORD'] if ENV['TARGET_PASSWORD']
options[:auth_methods] = %w[password publickey]set :host, options[:host_name] || ENV['TARGET_HOST']
set :ssh_options, options
We exclude tests not applicable for non-QEMU or non-VirtualBox via RSpec exclusion filters (read more at https://relishapp.com/rspec/rspec-core/docs/filtering/exclusion-filters)
is_virtualbox = ENV['BUILDER_TYPE']&.start_with?('virtualbox')
is_qemu = ENV['BUILDER_TYPE'] == 'qemu'config.include RSpec::GoldenFiles
config.filter_run_excluding is_virtualbox: !is_virtualbox
config.filter_run_excluding is_not_virtualbox: is_virtualbox
config.filter_run_excluding is_qemu: !is_qemu
config.filter_run_excluding is_not_qemu: is_qemu
end
Base-debian test file and how we mirror provision scripts with “mixin” tests
Our test is simply
# frozen_string_literal: truerequire 'spec_helper'describe 'base debian image' do
include_examples 'virtualbox guest additions'
include_examples 'qemu guest agent'
include_examples 'SSH'
include_examples 'disable lvmetad'
include_examples 'logging'
end
This structure of extracting provision script test logic to shareable examples (read more at https://relishapp.com/rspec/rspec-core/docs/example-groups/shared-examples) allows us to change as fast and easy as you can remove provisioners from your Packer template.
Let’s look at our first included example “virtualbox guest additions” .
It needs to test the following provision script:
#!/usr/bin/env bash
echo "Installing VirtualBox guest additions"apt install -y linux-headers-$(uname -r) build-essential
apt install -y dkmsVBOX_VERSION=$(cat /root/.vbox_version)
mount -o loop /root/VBoxGuestAdditions_$VBOX_VERSION.iso /mnt
sh /mnt/VBoxLinuxAdditions.run
umount /mnt
rm /root/VBoxGuestAdditions_$VBOX_VERSION.iso
The test file looks like (relevant ServerSpec resource docs at https://serverspec.org/resource_types.html)
# frozen_string_literal: truerequire 'spec_helper'RSpec.shared_examples 'virtualbox guest additions' do
context 'when running in virtualbox guest', is_virtualbox: true do
context 'with dependencies' do
describe package('build-essential') do
it { is_expected.to be_installed }
enddescribe package('dkms') do
it { is_expected.to be_installed }
enddescribe 'package "linux-headers"' do
it do
kernel_version = command('uname -r').stdout.strip
pkg = package("linux-headers-#{kernel_version}")
expect(pkg).to be_installed
end
end
enddescribe file('/root/.vbox_version') do
it { is_expected.to exist }
enddescribe kernel_module('vboxsf') do
it { is_expected.to be_loaded }
end
endcontext 'when not running in virtualbox guest', is_not_virtualbox: true do
describe kernel_module('vboxsf') do
it { is_expected.not_to be_loaded }
end
end
end
It’s important to note the is_not_virtualbox: true and is_virtualbox: true “tags” that are used for exclusion if it’s QEMU — we’re not going to test the is_virtualbox case at all. We’re only going to test the is_not_virtualbox one. For VirtualBox — the opposite.
And let’s see an example with a golden file, verifying that included example SSH matches the desired behaviour and guest state.
Provision script: (read more about PACKER_HTTP_ADDR at https://www.packer.io/docs/provisioners/shell.html#packer_http_addr)
#!/usr/bin/env bash
echo "Configure SSH keys"
apt update
apt install -y wgetwget http://${PACKER_HTTP_ADDR}/root_rsa.pub -O /tmp/root_rsa.pub
cat /tmp/root_rsa.pub >> /root/.ssh/authorized_keys
wget http://${PACKER_HTTP_ADDR}/users_rsa.pub -O /tmp/users_rsa.pub
cat /tmp/users_rsa.pub >> /root/.ssh/authorized_keys
root_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmauj4TaqalXVBnrMPSCaWx47EWqw7VE7H4fxp78YqUaNzhz1Y7f+BVmGCVCzvLMWF5PK+Yx66QuOu/R1xVRoyZRMn19QR/Roy7SKKbPT5GR//MP9NM9X78safSs318CRPsKKkkz5N2Qjv0nl5KzUtg6X3oKhgEptRIMnMu5y6lj2MxVxtKPVDLBm/vFiR6AzpN8/0/gD/mOgYpwbj0eOUZUy4I+HCovTFqQDhredvaS2h5ugyRL9EsoFSVVbgkdtNpk1HKjNf1bI0dEELgCxGlgG2cdnQXOYrtFX+Q3+0Y/bv3ruO43GsI88QuByHNR4okUnEkb0svQhQg2X8Z2fP root@example.comusers_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJJO1eEqDdRf2sJRt6cuaKxDYxfO4rU4EL4Q29zaCuc4l/YVzGKasmq43griT5QbmYS4K+xgVxwAgfgBZ7CsqQuv85KoJqjZFAjehptEUUtiIeZzvFTKd2nsfGzoR+UGzI5hjyJIglzilpqpUL2FseQzHFBne66kokzv1vAMe6kBMOaqflw1CaurAP6R64sAsIwaLRjmaCzpoWyFCjy4m4Lj5LV41GGHusDt3ASJnZwwTWPKxPH6BMlWgCWmHH6y8sqVxLyOaJYnXXJE3hdnxdbJ2UWgigJEjHakFU2HGPF0AWJV585xG1fY89U635fsMZ2lFThKIDN367KUTPz6Sb users@example.comThen our test file looks like
# frozen_string_literal: truerequire 'spec_helper'RSpec.shared_examples 'SSH' do
describe package('wget') do
it { is_expected.to be_installed }
enddescribe 'contents of authorized_keys' do
it do
guest_file = file('/root/.ssh/authorized_keys').contentexpect(guest_file).to match_golden_file('spec/goldenfiles/ssh/authorized_keys')
end
end
end
We use match_golden_file to match the following golden file
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmauj4TaqalXVBnrMPSCaWx47EWqw7VE7H4fxp78YqUaNzhz1Y7f+BVmGCVCzvLMWF5PK+Yx66QuOu/R1xVRoyZRMn19QR/Roy7SKKbPT5GR//MP9NM9X78safSs318CRPsKKkkz5N2Qjv0nl5KzUtg6X3oKhgEptRIMnMu5y6lj2MxVxtKPVDLBm/vFiR6AzpN8/0/gD/mOgYpwbj0eOUZUy4I+HCovTFqQDhredvaS2h5ugyRL9EsoFSVVbgkdtNpk1HKjNf1bI0dEELgCxGlgG2cdnQXOYrtFX+Q3+0Y/bv3ruO43GsI88QuByHNR4okUnEkb0svQhQg2X8Z2fP root@example.com
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJJO1eEqDdRf2sJRt6cuaKxDYxfO4rU4EL4Q29zaCuc4l/YVzGKasmq43griT5QbmYS4K+xgVxwAgfgBZ7CsqQuv85KoJqjZFAjehptEUUtiIeZzvFTKd2nsfGzoR+UGzI5hjyJIglzilpqpUL2FseQzHFBne66kokzv1vAMe6kBMOaqflw1CaurAP6R64sAsIwaLRjmaCzpoWyFCjy4m4Lj5LV41GGHusDt3ASJnZwwTWPKxPH6BMlWgCWmHH6y8sqVxLyOaJYnXXJE3hdnxdbJ2UWgigJEjHakFU2HGPF0AWJV585xG1fY89U635fsMZ2lFThKIDN367KUTPz6Sb users@example.comWe’re going to skip reviewing the other test suite files, cause they won’t be that much different.
Executing the test suite
Since we chose to get the SSH IP and port from the packer log file, we must always run our tests with PACKER_LOG=1 and PACKER_LOG_PATH=packer.log (and `packer.log` must be in the same dir as the packer template).
Run the following command to create guest image with tests before exporting it:
PACKER_LOG=1 PACKER_LOG_PATH=./packer.log packer build \
-var 'must_run_tests=true' \
-var 'headless=true' \
-var 'host_adapter=vboxnet0' \
-only=virtualbox-base-debian \
-force packer-base-debian.jsonWe must see output very similar to the one below with a successful outcome
virtualbox-base-debian: base debian image
virtualbox-base-debian: when running in virtualbox guest
virtualbox-base-debian: with dependencies
virtualbox-base-debian: Package "build-essential"
virtualbox-base-debian: should be installed
virtualbox-base-debian: Package "dkms"
virtualbox-base-debian: should be installed
virtualbox-base-debian: package "linux-headers"
virtualbox-base-debian: should be installed
virtualbox-base-debian: File "/root/.vbox_version"
virtualbox-base-debian: should exist
virtualbox-base-debian: Kernel module "vboxsf"
virtualbox-base-debian: should be loaded
virtualbox-base-debian: when not running in qemu guest
virtualbox-base-debian: Package "qemu-guest-agent"
virtualbox-base-debian: should not be installed
virtualbox-base-debian: Package "wget"
virtualbox-base-debian: should be installed
virtualbox-base-debian: contents of authorized_keys
virtualbox-base-debian: should be matching golden file with name spec/goldenfiles/ssh/authorized_keys
virtualbox-base-debian: /etc/lvm/lvm.conf
virtualbox-base-debian: should contain "use_lvmetad = 0"
virtualbox-base-debian: File "/var/log/journal"
virtualbox-base-debian: should be exists
virtualbox-base-debian: File "/etc/systemd/journald.conf"
virtualbox-base-debian: should be matching golden file with name spec/goldenfiles/logging/journald.conf
virtualbox-base-debian: Service "systemd-journald"
virtualbox-base-debian: should be enabled
virtualbox-base-debian:
virtualbox-base-debian: Finished in 3.56 seconds (files took 0.18682 seconds to load)
virtualbox-base-debian: 12 examples, 0 failures
virtualbox-base-debian:
==> virtualbox-base-debian: Gracefully halting virtual machine...
==> virtualbox-base-debian: Preparing to export machine...
virtualbox-base-debian: Deleting forwarded port mapping for the communicator (SSH, WinRM, etc) (host port 3462)
==> virtualbox-base-debian: Exporting virtual machine...
virtualbox-base-debian: Executing: export packer-virtualbox-base-debian --output output-virtualbox-base-debian/packer-virtualbox-base-debian.ova
==> virtualbox-base-debian: Deregistering and deleting VM...
Build 'virtualbox-base-debian' finished.==> Builds finished. The artifacts of successful builds are:
--> virtualbox-base-debian: VM files in directory: output-virtualbox-base-debian
Debugging the test suite
We’ve also added a provisioner that sleeps just before the test suite
{
"type": "shell-local",
"script": "./serverspec/sleep_before_tests_packer.sh",
"environment_vars": [
"MUST_SLEEP_BEFORE_TESTS={{ user `must_sleep_before_tests` }}"
]
},It just does sleep 99999 so that you can develop and extend the test suite without the machine existing on you and Packer removing the SSH port forwarding.
The above provisioner can be run using
PACKER_LOG=1 PACKER_LOG_PATH=./packer.log packer build \
-var 'must_sleep_before_tests=true' \
-var 'headless=true' \
-var 'host_adapter=vboxnet0' \
-only=virtualbox-base-debian \
-force packer-base-debian.jsonWhere to go from here
You have a good idea what we do in SumUp to test our images created via Packer.
We have a bit more complex image hierarchies and test suites, but they’re expanded upon this simple one.
You can use this one as a template.
Again, the source code is at https://github.com/syndbg/sumup-blog-hashicorp-packer-test
Thank you for your time, Anton.
