Demystify how Gitpod handles Git authentication, mistify your Docker setup

Containerizing Git Credential Helpers

How to expose Git credential helpers to containerized processes, allowing the use of Bitbucket, Github, and Gitlab inside of Docker on Gitpod.

Git credential helpers (image source)

Pulling or pushing from Git often requires authentication. Typically, you do this using a username and password or SSH keys. When working with Gitpod, authentication somehow works out of the box.

This story will examine how Gitpod and similar services perform their Git authentication magic. We learn about Git credential helper processes and how to interface with them. We will also look at using named pipes, also known as FIFOs, to communicate information (such as Git credentials) between containerized processes and their host system.

Why containers?

Thanks to credential processes and a trust relationship you put in place between Gitpod and your Gitlab, Bitbucket, and/or Github accounts, Gitpod can make Git work with these services work out of the box. Unfortunately, this integration does not extend to Docker containers or containerized build processes within a Gitpod workspace.

Locally and when using SSH, you typically make host credentials available to your container by mounting your SSH agent’s socket into the running container or by mounting your local credentials during builds using BuildKit’s --ssh mount option. For more information on this, you can read this excellent blog post. When working with Gitpod, however, this is not an option — because from where do your “local” credentials originate? We look at how to make the credential processes available to your containerized applications.

Git authentication on Gitpod

We start by investigating how exactly Gitpod (and similar services) use Git credential helpers to authenticate you with your Git service of choice.

Authorization delegation

When logging in to Gitpod, you use your Github, Gitlab, or Bitbucket account(s). When doing this, you not only authenticate yourself with Gitpod; you also link Gitpod to your Git service of choice, delegating the authority that you carry with these services to Gitpod. Gitpod uses this delegated authority to make temporary Git credentials available within your workspace.

Helper application

To make Git credentials available, Gitpod mounts an application, gp, in every Gitpod workspace. It then registers this application as a Git credential helper by creating a ~/.gitrc file containing the following section:

[credential]
helper = /usr/bin/gp credential-helper

Git credential helpers are applications implementing a well-defined specific interface documented on the Git website. Credential helpers read some information from standard input (stdin) describing the repository for which to request credentials. The helper returns a suitable username and (usually temporary) password to standard output (stdout).

Note that you usually manipulate ~/.gitrc using calls to git:

git config --global credential.helper /usr/bin/gp credential-helper

Trying it out

The command gp credential-helper is a Git credential helper. Verify that running the following command returns Bitbucket credentials:

echo -e "protocol=https\nhost=bitbucket.org\n" \
| gp credential-helper get

If you do not use Bitbucket, change this command to request credentials for your provider of choice. You should see something like this:

protocol=https
host=bitbucket.org
username=x-token-auth
password=verysecrettemppassword

Authorizing containerized processes

Now that we know how Gitpod performs its authentication magic 🪄, we can turn our attention to making the same process available within containers 🐳.

Host networking

The most straightforward approach to making Git authentication work inside of containers would be to copy or mount the gp application into the container, and then register it as a credential helper.

Unfortunately, this approach is problematic. Under the hood, gp performs a call to a REST service running on ::1 ; this call will only end up in the right place (the host system) when using host networking (--net=host). Using host networking is not usually something we want to do. Internally, gp also appears to rely on curl being available; if curl is not installed within your container, gp will not present you with an error and hang.

Volume-mounting credentials

There are many alternative ways of getting your credentials to your container. The easiest one is probably to create a temporary credential file and mount it in the container. Since credentials expire, this may not be ideal. Currently, my favourite approach is to make Git credentials available to running containers not as static files but as named pipes (aka FIFOs). The complete process looks as follows:

• Run processes on the Gitpod host that write (temporary) Git credentials to named pipes.
• Mount these pipes into the container (using volume mounts) or build process (using BuildKit’s support for secret mounts).
• In the container, register a credential helper that cat s the appropriate credential file (FIFO).

Of course, you can automate this using startup scripts and aliases.

You can create the named pipes and spawn the processes that write to them by running a script like this:

Preferably, run this script in the background as a service when your Gitpod instance starts.

Running the script makes pipes available in the folder /var/run/gc. If you run the command:

ls /var/run/gc

You should see

bitbucket.org  github.com  gitlab.com

Verify that reading from the pipes yields a valid Git credential helper response by running cat /var/run/gc/bitbucket.org:

protocol=https
host=bitbucket.org
username=x-token-auth
password=verysecrettemppassword

If you do not use Github, Bitbucket and/or Gitlab, leave out some of the calls to serve.

Note that if you have some time to spare and a desire to challenge yourself, you could implement a FUSE driver instead of using FIFOs; this would likely be more efficient and would make it easier to understand when the credentials are created. If you do this, let me know in the comments!

Credentialed containers

To start a credentialed container, volume-mount the named credential pipes in the container when you start it:

docker run -it --rm -v /var/run/gc:/var/run/gc:ro ubuntu bash

Inside the container, register a script that reads from the appropriate credential file as a Git credential helper:

git config --global credential.helper '! f() { test "$1" = get && cat /var/run/gc/$(cat - | grep host | cut -d= -f2); }; f'

In the context of Git credential helper configuration, the exclamation mark means “interpret what follows as a script”.

If you only need access to a single Git provider, you can define the credential helper as a more straightforward cat statement. For example, authenticate to Bitbucket using the following helper:

git config --global credential.helper \
  '! cat /var/run/gc/bitbucket.org'

Now, verify that you have access to private Git repos!

git clone https://bitbucket.org/secretiveorg/verysecret.git

Credentialed builds

Run your build using BuildKit and mount the appropriate secrets:

DOCKER_BUILDKIT=1 docker build \
  --secret id=git,src=/var/run/gc/bitbucket.org

During a build, you can define a RUN statement with Git access as:

RUN --mount=type=secret,id=git \
git config --global credential.helper '! cat /run/secrets/git' && \
git clone https://bitbucket.org/secretiveorg/verysecret.git

To make your Dockerfile compatible with both SSH key mounting and our new custom credential helper, you can request to mount both and set the credential helper configuration conditionally:

RUN --mount=type=secret,id=git --mount=type=ssh \
{ [ ! -f "/run/secrets/git" ] \
&& git clone git@bitbucket.org/secretiveorg/verysecret.git ; } \ 
|| { git config --global credential.helper '! cat /run/secrets/git' \
&& git clone https://bitbucket.org/secretiveorg/verysecret.git ; }

Configuration using environment variables

Running credentialed containers using the above approach requires executing a command inside the container. It is possible to configure Git to use a custom prompt by setting the environment variable GIT_ASKPASS. Thanks to this, we can eliminate the need to run the git config command inside the container. The trick is to instruct Git to use a custom prompt that knows the answers to Git’s questions.

One way to do this is to define a script,prompt.sh, as follows. Put it together in the directory with the named pipes. It is convenient to create prompt.sh from the same script that creates the named pipes (like this).

Now, start credentialed containers as follows:

docker run -it --rm \
  -v /var/run/gc:/var/run/gc:ro \
  --env GIT_ASKPASS=/var/run/gc/prompt.sh \
  ubuntu bash

Git authentication now … works! If you want real magic 🎩, go the extra mile (or, you know, another extra mile) by adding Gitpod configuration that adds this -v and --env argument automatically to calls to docker run.

Using the GIT_ASKPASS approach during builds is inconvenient. You have to either create the named pipes and prompt.sh file in a folder inside the build context and bind-mount or define prompt.sh as a (second) secret.

Considerations

The solutions above work quite well, but making credential helpers available inside containerized projects is non-trivial. More often than not, this complexity will not be worth it.

Consider whether you need credentialed access to Git within containers. Can you check out the repo on the host and mount the result? Is it not an option to use SSH keys or to “manually” authenticate? Think about your needs before jumping into implementing the above.