Using Pack with a remote Docker daemon

Published in

Buildpacks

7 min readSep 1, 2020

Do you need to create a Docker image but don’t have Docker running on your machine? No problem. Pack uses the Docker API to run containerized builds and create runnable OCI images, which means it works equally well with a secure remote daemon as it does with a local daemon.

Using a remote daemon looks something like this:

export DOCKER_HOST=tcp://daemon-hostname:2376
export DOCKER_CERT_PATH=~/.docker/
export DOCKER_TLS_VERIFY=1pack build my-app —-path ./my-sourcedocker run --publish 8080:80 my-appcurl http://daemon-hostname:8080

To try it, you’ll need a secure remote daemon of your own. The quickest way is using a Docker in Docker daemon, which simulates a remote daemon by running one locally in a container:

# Start a Docker daemon in a background container
docker run -d --privileged --name dind \
  -e DOCKER_TLS_CERTDIR=/certs \
  -v $PWD/certs:/certs \
  --publish 2376:2376 \
  docker:dind# Set Docker API settings
export DOCKER_HOST=tcp://localhost:2376 \
       DOCKER_CERT_PATH=$PWD/certs/client \
       DOCKER_TLS_VERIFY=1# Build a sample app with pack
git clone https://github.com/buildpacks/samplespack build my-app --path samples/apps/bash-script/ --builder cnbs/sample-builder:alpine# Run it with docker
docker run my-app

You’ll see some satisfying ASCII art!

But that daemon wasn’t truly remote, nor all that useful in production. Now we’ll go through the steps needed to secure a real, remote Docker daemon.

“Cargo ship on the Rhine — Riehl, Cologne” by marcoverch is licensed under CC BY 2.0

Why use a Remote Docker?

Sometimes you can’t install Docker locally. Or maybe you need to develop against different versions, operating systems, or distributions of the many Docker options and alternatives.¹

Docker clients (usually docker CLI) and daemons (usually dockerd)use the same Docker API whether communicating locally or remotely. Local connections are either via a socket or insecure TCP tolocalhost, and remote connections via TLS+TCP.

The pack CLI is also a client, using the same Docker API and Docker’s excellent client library. It runs on MacOS, Linux and Windows — any of which can build Linux or Windows images depending on the operating system of its connected Docker daemon.

Running these clients and daemons separately can let a low-privileged client send commands and source code to any daemon to build, execute, push/pull and manage volumes/bind-mounts.

Start local

To better understand how Docker clients and daemons can be configured, here’s a local-only example of a docker daemon using both sockets and insecure TCP at the same time:

sudo dockerd -H unix:///var/run/docker.sock -H tcp://127.0.0.1:2375

A docker client on the same machine can communicate either way:

export DOCKER_HOST=unix:///var/run/docker.sock   # default
docker infoexport DOCKER_HOST=tcp://localhost:2375
docker info

And so can pack:

export DOCKER_HOST=unix:///var/run/docker.sock   # default
pack build my-app —-path ./my-sourceexport DOCKER_HOST=tcp://localhost:2375
pack build my-app —-path ./my-source

It’s the DOCKER_HOST environment variable that directs a client to a different daemon listener.

This setup is also useful if you want to experiment with the Docker API yourself. You can curl localhost:2375 and interact directly with many endpoints of your daemon:

curl http://localhost:2375/containers/json | jq

Running Docker securely, remotely

Local daemons are handy but if you can’t run the daemon you want locally, you’ll need to make it available on your network. Docker has only one built-in configuration for secure, authenticated client/daemon interaction: Mutual TLS over TCP.²

Here’s an example of a daemon configured to use mutual TLS certificates and listen on the standard secure docker port 2376:

sudo dockerd -H tcp://0.0.0.0:2376 \
  --tlsverify \
  --tlscacert="/etc/ssl/docker/ca.pem" \
  --tlscert="/etc/ssl/docker/server-cert.pem" \
  --tlskey="/etc/ssl/docker/server-key.pem"

You’ll need certs of course, which we’ll get to shortly. But from your workstation or another machine, you’d connect to it like this:

# on your workstation
export DOCKER_HOST=tcp://daemon-hostname:2376
export DOCKER_CERT_PATH="~/.docker/"   # has cert.pem,key.pem,ca.pem
export DOCKER_TLS_VERIFY=1docker infopack build my-app —-path my-source

Now when it comes to running your app, remember it’s running on your daemon’s machine, not your workstation, so you’ll have to reach your app over the network too instead of usinglocalhost³:

docker run --publish 8080:80 my-appcurl http://docker-hostname:8080

This also changes volume path mounting — you can only mount volumes and paths on your daemon’s machine. However, your app source still comes from your workstation:

# docker using bind-mount volumes
docker build ./my-local-source --tag my-docker-appdocker run --volume /path-on-daemon-host:/my-mount \
  my-docker-app ls -l /my-mount# pack using bind-mount volumes
pack build my-pack-app \
  --volume /path-on-daemon-host \
  --path ./my-local-source

In both examples, ./my-local-source will be copied over the network from your workstation but not /path-on-daemon-host, which must be an existing directory on your daemon’s machine.

Doing it all wrong

The easiest mistake to make is configuring your daemon to run with insecure TCP on a shared network — roughly equivalent to granting password-less root access to your daemon’s machine for everyone on your network.

I won’t show a copy-pwnable example of doing it wrong but never run dockerd on an untrusted network with a TCP listen address like 0.0.0.0:2375 and no corresponding --tlsverify option.

Please don’t feed the Bitcoin miners.

Certs are key

But where do all these magical certificates come from?

A properly secured Docker daemon uses Mutual TLS to ensure only authorized clients can talk to your daemon (and visa versa). This is all enforced with standard x509 keys and certs using familiar PKI tools. The Docker-in-Docker example at the beginning generated them automatically but here’s a simple, self-signed way to do it yourself:

# Set to a hostname you can access from your workstation
# Use a valid DNS name or /etc/hosts entry
HOSTNAME=daemon-hostname# Certificate Authority cert
openssl req -new -x509 -days 365 -sha256 -newkey rsa:4096 -nodes \
  -subj "/C=ZZ/ST=ZZ/L=ZZ/O=ZZ/CN=$HOSTNAME" \
  -out ca.pem -keyout ca-key.pem# Server cert
openssl req -new -newkey rsa:4096 -nodes \
  -subj "/CN=$HOSTNAME" \
  -out server.csr -keyout server-key.pem 
echo subjectAltName = DNS:$HOSTNAME,IP:127.0.0.1 > extfile.cnf
echo extendedKeyUsage = serverAuth >> extfile.cnf
openssl x509 -req -days 365 -sha256 -extfile extfile.cnf \
  -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
  -out server-cert.pem# Client cert
openssl req -subj '/CN=client' -new -newkey rsa:4096 -nodes \
  -out client.csr -keyout key.pem
echo extendedKeyUsage = clientAuth > extfile-client.cnf
openssl x509 -req -days 365 -sha256 -extfile extfile-client.cnf \
  -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
  -out cert.pem# Cleanup temp files
rm -v client.csr server.csr extfile.cnf extfile-client.cnf

This generates the following certs and keys:

A new self-signed certificate authority CA cert/key
A new server cert/key with it’s hard-coded DNS hostname⁴
A new client cert/key that can be used by anyone

Clients, Daemons ❤️ Certs

Now that your certs are generated, you need to put them in the right places so your client and server can use them:

Move cert.pem, key.pem and ca.pem to your workstation’s default client cert location in your docker settings folder and set secure permissions.

# on workstation
mv cert.pem key.pem ~/.docker
cp ca.pem ~/.dockerchmod 0400 ~/.docker/key.pem
chmod 0444 ~/.docker/cert.pem ~/.docker/ca.pem

Move server-cert.pem ,server-key.pem and copy ca.pem to a standard cert location on your daemon host:

# on daemon host
mv server-cert.pem server-key.pem /etc/ssl/docker/
cp ca.pem /etc/ssl/dockerchmod 0400 /etc/ssl/docker/server-key.pem
chmod 0444 /etc/ssl/docker/{server-cert.pem,ca.pem}

Don’t forget to back up your Certificate Authority certs to a safe place so you can generate more compatible certs later on:⁵

# backup
mv ca-key.pem ca.pem /safe-place

Now, configure your daemon to use your new certs. There’s a few options:

Option A) Change the command line:

sudo dockerd -H tcp://0.0.0.0:2376 \
  --tlsverify \
  --tlscacert="/etc/ssl/docker/ca.pem" \
  --tlscert="/etc/ssl/docker/server-cert.pem" \
  --tlskey="/etc/ssl/docker/server-key.pem"

Option B) Edit the daemon config /etc/docker/daemon.json:

{
...
"tls": true,
"tlsverify": true,
"tlscacert": "/etc/ssl/docker/ca.pem",
"tlscert": "/etc/ssl/docker/server-cert.pem",
"tlskey": "/etc/ssl/docker/server-key.pem",
...
}

There’s more information on configuring your Docker daemon in Docker’s docs.

Restart your daemon and test it all out from your workstation!

export DOCKER_HOST=tcp://daemon-hostname:2376
export DOCKER_CERT_PATH=~/.docker/
export DOCKER_TLS_VERIFY=1pack build my-app —-path ./my-sourcedocker run --publish 8080:80 my-appcurl http://daemon-hostname:8080

Congratulations! You have a secure Docker daemon, accessible only to you, but runnable just about anywhere. Now you can run pack , docker or just about any Docker API-capable client, just like running locally.

Try this at home — or in the cloud

Here’s a few ways to try out this configuration on a new machine:

And some other ideas you might explore with your mTLS-encrypted Docker:

Generate a client cert for a collaborator to share your daemon host⁵
Troubleshoot containers on remote Docker-based CI instances
Build a dashboard using the Docker API to monitor a remote daemon

Learn more

Pack and buildpacks:

Quickstart and Docs: https://buildpacks.io/docs
More on Medium: https://medium.com/buildpacks

Authorization options for Docker daemons:

Authorization plugins

More about Mutual TLS

The magic of TLS, X509 and mutual authentication explained

Footnotes

[1] The Docker API has a ton of features and not all Docker implementations provide those needed by pack. If there’s an implementation you really want that doesn’t work yet, please file an issue (or a PR!)

[2] Docker does support authentication without authorization when used with --tls but without --tlsverify. This should never be used without an authorization plugin or alternative authentication mechanism.

[3] Published ports on a remote daemon are not automatically secure. Make sure your apps themselves are appropriately secure to run on the network.

[4] This example generates certs for using DOCKER_HOST with your daemon machine’s DNS name (i.e. tcp://daemon-hostname:2376). If you really want to use an IP address instead (i.e. tcp://10.0.0.2:2376), just insert your daemon machine’s IP to this line in the example:

echo subjectAltName = DNS:$HOSTNAME,IP:10.0.0.2 > extfile.cnf

[5] To generate a new client cert, just repeat the Client or Server sections of the openssl example. For a new server cert, adjust the subjectAltName settings for your new server’s name/IP.