Building custom machine images with Packer

In this article we shall build a custom machine image for the google cloud platform. This article is not exhaustive of all platforms packer can build an image for; there’s tonnes of platforms it builds images for as can be seen from its website packer.io.

With that said, let’s delve into building the images for the above mentioned platform.

Google Cloud Platform

For packer to build a custom image, it needs a “blueprint”so as to know what to include in the image. The “blueprint” is the packer template and is put together by you. It is a *.json file which tells the packer builder what cloud provider to build for, what softwares to install in your custom built image, and much more as we are yet to see.

Let’s quickly put together the packer template. I’m going to name it “packer.json” but it can be named anything, as long as it is a .json extension.

Variables

“variables”: {
“service_account_json”: “path/to/authentication/file”,
“repo_path” : “path/to/local/or/remote/code/repository”,
“project_id”: “{{env `PROJECT_ID`}}”
}

The above code block is used to set any variables that the packer template will need while building our custom image. This is the section we define variables such as, authentication credentials required for packer and google cloud to interact. This is needed when packer is building the image as it interacts with the Google cloud to start an instance, do all the necessary setup as described in the packer template, and then later create the image and store it on Google cloud itself.

while packer is running in your terminal, head over to the Google Cloud console, under the Google Compute Engine section, you will notice an instance that was not there previously and won’t be after packer is run

The above key-value pairs in the variables code block, in this case, are the google service account credentials (service_account_json) which is a json file, the directory path (repo_path) to the local repository of the code you would like to bake into the resulting packer image which is set as an environment variable(this variable is only necessary if the resulting image will be used to setup your application server otherwise it can be skipped), and the google project ID (project_id), a unique identifier for the google project we are building the custom image for.

We could use a file path for all the above variables, but it’s best practice to use environment variables instead.

Builders

“builders”: [
    {
      “type”: “googlecompute”,
      “project_id”: “{{user `project_id`}}”,
      “machine_type”: “n1-standard-1”,
      “source_image”: “ubuntu-image”,
      “region”: “europe-west1”,
      “zone”: “europe-west1-b”,
      “image_description”: “custom machine image”,
      “image_name”: “new_custom_image”,
      “disk_size”: 10,
      “account_file”: “{{ user `service_account_json`}}”
    }
]

The above code block is an array of one or more components that defines the builders that will be used to create the machine image from this template, and configures each of those builders. For our case here, we are using the googlecompute builder.

In the builders code block, we have:

• Type (type) key/value pair. This is the name of the builder that will be used to create the custom machine image for the current packer build.
• Project ID (project_id) key/value pair. This is the google cloud project identifier that packer uses to identify which project it interacts with while building the custom image. Additionally it is used for authentication purposes.
• Machine type (machine_type) key/value pair. This is the machine type you intend to use while you build the custom image. The machines are of various types you can choose from. Selection of a machine type depends on your setup needs and requirements. You can find all the machine types for google cloud platform here.
• Source image (source_image) key/value pair. This is the image whose base Operating System you intend to build on top of. For this article’s sake, this can be ubuntu or windows or whatever Operating System you intend to build on top of. Building on top of an already baked image saves us a lot of build time as you don’t have to install the operating system for this custom image we are building. The source image comes with most of the OS specific packages and softwares pre-installed.
• Region and zone (region, zone) key/value pairs. These are the regions and zones the machine image will run in. Those specified here are google cloud specific, but other cloud providers have equivalents for the same.
• Image description (image_description). This is the description of what the image you’re building is all about. This provides information about the image created.
• Image name (image_name). This is the name of the custom image you are building. This name should be unique.
• Disk size (disk_size). This is the size in Gigabytes of the volume your image should reserve. This can be any number above zero, as long as you can pay for it given the cloud provider billing. Best advice would be you check the provider billing for any resources you intend to use before you use it as that will keep your expenditure in check.
• Service account json file (account_file). As mentioned above in the variables block section, this is for authentication purposes with the cloud provider you’re building an image for. In this code block, we reference the account_file key we declared above in the variables code block whose value is the authentication credentials json file. You may be wondering where this file comes from, or how to write it. The answer is, you don’t write it, google generates it for you as you create a service account for your google project.

There’s more arguments that can be added to the template depending on your configuration. All possible arguments that can be included can be found here

Provisioners

“provisioners”: [ {
        “type”: “file”,
        “source”: “path to file”,
        “destination”: “file/path/to/destination folder/in/the/new/machine/image”
    },
    {
        “type”: “shell”,
        “script”: “bash_script.sh”
    },
    {
        “type”: “file”,
        “source”: “{{ user `service_account_json`}}”,
        “destination”: “/path/to/file/account.json”
     } ]

The provisioners code block contains an array of all provisioners that Packer will use to install and configure software within the custom image that is being created and eventually the same software will be present in any of the instances created from this image. Some of the roles of provisioners include, but not limited to:

• installing packages
• patching the kernel
• creating users
• downloading application code

Provisioners are optional. Hence, if no provisioners are defined within a packer template, then no software other than the defaults will be installed within the resulting custom machine image.

There is a number of provisioners out there as you can find here, but for this article I will only illustrate the ones we shall be using for this demo.

The first provisioner is of type file. The role of this is to upload files to the machine image that is being built by packer for later use by any service that may need them and will be available in all instances created from this image. These files may contain authentication tokens, application code, etc.

If need be, you can still upload an entire file folder using this same provisioner. You could look into how to do that here.

As we define the file provisioner, we also have to define the source and destination of the file we intend to upload using the source and destination key/value pairs respectively. The source is where the file or folder is on the local system you are running packer on, and the destination is where you want the file or folder to reside, in the new machine image you are building, in each and every instance created from this image.

The other provisioner used, is the shell provisioner. The role of this provisioner is to execute shell commands just like you would normally do within the computer terminal. These shell commands install softwares into the machine image we are building with packer. You can specify as many shell provisioners as you want for all the commands you want to run while building the image. But it is best practice that you combine most of the similar commands (commands that are dependant on each other or related in functionality) into a bash script and execute the script instead with the shell provisioner. Using bash scripts this way makes debugging your packer template file a lot more easier.

More provisioners and how they can be used can be found here.

With that said, let’s assemble the various code blocks together to make a meaningful packer template and build the custom image.

Packer template

{
    “variables”: {
         “service_account_json”: “../shared/account.json”,
         “repo_path” : “{{env `REPO_PATH`}}”,
         “project_id”: “{{env `PROJECT_ID`}}”
     },
“builders”: [ 
        {
           “type”: “googlecompute”,
           “project_id”: “{{user `project_id`}}”,
           “machine_type”: “n1-standard-1”,
           “source_image”: “ubuntu-image”,
           “region”: “europe-west1”,
           “zone”: “europe-west1-b”,
           “image_description”: “custom machine image”,
           “image_name”: “new_custom_image”,
           “disk_size”: 10,
           “account_file”: “{{ user `service_account_json`}}”
        }
     ],
“provisioners”: [
         {
             “type”: “file”,
             “source”: “path to file”,
             “destination”: “file/path/to/destination folder/in/the/new/machine/image”
         },
         {
             “type”: “shell”,
             “script”: “bash_script.sh”
         },
         {
             “type”: “file”,
             “source”: “{{ user `service_account_json`}}”,
             “destination”: “/path/to/file/account.json”
          }
      ]
}

Before we run the packer build command to finally build the custom image, make sure that packer is installed and the environment variables are set. The environment variables that we declared in the variables block are the ones whose values we have to set. If we do not set them, packer will “complain”. These are set the same way we set normal environment variables via the computer terminal.

With the environment variables set, we can now run packer by running this command in the terminal packer build packer.json.

Remember we named our packer template file packer.json

When packer is done building the image, head over to the GCP console, in the images section, and viola, the newly created image will be there. See the image below for reference.

And with that, an image has successfully been created for Google Cloud Platform. Instances can be created from this image just like any other image google provides by default.

That’s it for this article, for the next one in this series, I will take you through building a machine image for another provider.

For any additions and/or questions, feel free to post them here or reach out directly on twitter. Adios!