Flexible docker images with optional COPY

By: Yongjie Lim

We use a general-purpose base image for all our Rails apps here at TrueCar. By leveraging the ONBUILD directive to do common setup tasks, we obviate the need for developers to copy + paste those common tasks in their Dockerfiles. Recently, the ability of our general purpose image to handle specific app dependencies was tested, leading us to our topic today: how to specify optional files in the COPY Docker directive to satisfy app specific requirements.

Some Background

Our base image handles a number of setup steps crucial to the successful launching of a Rails app; chief among those is the installation of gems and their system dependencies. Handling these steps in the base image means that developers don’t need to worry about installing (and removing) the build dependencies in their Dockerfiles, or worry about Docker cache optimization with regards to bundle install.

The Problem

Until the particular occurrence detailed in the next paragraph, our one-dependency-list-fits-all assumption worked fine for the multitude of Rails apps we support in the company. App specific dependencies were either small enough that we could simply add it to the list of build dependencies we supplied to every app, or needed only for runtime and not for building gems.

One of our major apps required a large, app-specific gem dependency that could not be thrown onto the base image without needlessly expanding its size. The app developers tried to install the dependency in the app Dockerfile to no avail; because the base image’s ONBUILD directives fire before any steps defined in the app Dockerfile, gem installation runs and fails before the required dependency can be installed.

Docker Copy to the Rescue?

By including an onbuild COPY directive in the base image’s Dockerfile, we could provide a script file from the app to call during the onbuild step executions, like so:

ONBUILD COPY docker-extra-setup /tmp/docker-extra-setup
ONBUILD RUN /tmp/docker-extra-setup || echo ‘No extra setup provided.’


Step 1/1 : COPY docker-extra-setup /tmp/docker-extra-setup
Step 1/1 : RUN /tmp/docker-extra-setup || echo ‘No extra setup provided.’
---> Running in 985d1cf5c9ca
Extra setup would be executed here!
Step 1/1 : COPY Gemfile* ./

Easy right?

But what if the file isn’t provided by the app?

Step 1/1 : COPY docker-extra-setup /tmp/docker-extra-setup
lstat docker-extra-setup: no such file or directory

Maybe being explicit is the wrong way to do it. What if we just provided it a wildcard instead?

Step 1/1 : COPY *docker-extra-setup /tmp/docker-extra-setup
No source files were specified

Unfortunately, Docker COPY doesn’t like it when either the file it’s trying to copy is missing, or nothing is provided for it to copy. Without some way to specify that that file is optional, we’d have to include it any app that builds from the base-image — whether it’s used or not!

NOTE: Why not just rely on “COPY . .” to load all the app’s files, and not explicitly copy in the setup file? We want to take advantage of Docker cache to cache the gem installation step, so gem install has to happen before all the files are copied over (otherwise, any change to a file in the app would cause a subsequent bundle install step to not use the Docker cache!). It could be also added to a common directory all the apps use…but if that directory gets changed often, the same problem would happen.

Docker Optional Copy to the Rescue

Luckily, we can trick Docker into copying our setup file, whether it exists or not, by giving it what it wants: a file to copy over.

Setup file present:

Step 1/1 : COPY *docker-extra-setup Dockerfile /tmp/
Step 1/1 : RUN /tmp/docker-extra-setup || echo ‘No extra setup provided.’
---> Running in af2a02361f3b
Extra setup would be executed here!
Step 1/1 : COPY Gemfile* ./

Setup file absent:

Step 1/1 : COPY *docker-extra-setup Dockerfile /tmp/
Step 1/1 : RUN /tmp/docker-extra-setup || echo ‘No extra setup provided.’
---> Running in d53a235c3db4
/bin/sh: /tmp/docker-extra-setup: not found
‘No extra setup provided.’
Step 1/1 : COPY Gemfile* ./

By feeding COPY a file that we always know will be there, such as a Dockerfile at the root directory, we can satisfy its need to copy a valid file while having it overlook a potentially missing setup file.

That’s it! We hope that any of you out there experimenting with Docker find a useful application of this trick!