Running Docker builds with private Nuget repositories using Service Containers
Over the last few years, I’ve been continually evolving how to use private Nuget repositories in Azure Piplines. If you look at the massive GitHub issue at the Azure Pipelines Tasks repo, you can see how the solutions have evolved.
In my first attempt, I used the Nuget credential provider along with a short-lived access token as a Docker ARG to connect to Nuget. This worked well but had a flaw…. Because the token was getting passed in as a ARG, the dotnet restore step was not getting cached. If you have a large application, the ability to cache this step is vital to keeping build times down.
On try 2, I wrote about using Docker secrets along with BuildKit. This solution was nice because it was now possible to use the PAT without affecting the build cache. Unfortunately, --cache-from is not supported in BuildKit which means your images will be built from scratch every time if you are using a hosted Azure Pipelines agent. On a self-hosted agent, this isn’t an issue because the previous images are already on the agent so Docker will use them to build new ones. On a hosted agent though, --cache-from is important because it will allow you to specify an exact image to build on top of. I don’t want to deal with managing my own agents so I was back to square one at this point.
Recently, Azure DevOps released a feature called Service Containers. The basic premise of them is that your build will execute on an agent but you can define dependency containers. Your build can reference these containers and you can even run your build inside of one. This is useful if you have a set of tests that require external dependencies like Redis or Sql Server and you want to spin up an actual instance of those resources to test against.
If I could move Nuget authentication outside of my application build containers and connect to an “anonymous” Nuget server, caching would work again. I started researching Nuget proxies but interestingly enough, I couldn’t find any free or open source ones. I found an open source Nuget server called BaGet that allowed upstream package searching. This was enough for me to be able to move forward.
I cloned the repo and stripped out all of the pieces I didn’t need which became ThinNugetProxy (super creative name… I know). To support my functionality, I needed to support generating the URL manifest for all the features that Nuget provides but only needed to have endpoints for getting package versions and actually fetching the packages themselves. The last piece I needed to add was authentication for upstream sources.
Ideally, the Nuget proxy container would just take two runtime parameters; the Access Token and the upstream package source, and use that for all of its connections during the duration of the build. With Azure Artifacts being my upstream source, I was able to just use Basic authentication where the username is any string and the access token is the password.
After a quick local smoke test, it seemed like everything is working!
When using service containers, there are two routes you can go. If you are hosting your build inside of a service container, you’ll actually have the ability to call your other containers via their name. The other option which I went with is to map a port from the Agent to your container.
So now from my build, I should be able to use localhost:8080 as a Nuget server. Inside of my container though, you need to reach out to the host. On MacOs and Windows this is easy. There are DNS entries in the container that let you easily connect to the host. With those you can just use host.docker.internal which is always mapped to the hosts IP. Annoyingly enough though, this doesn’t work with Linux. I ended up just referencing the host by IP which seems to consistently be 172.17.0.1. The final line you use to restore in the Dockerfile is
RUN dotnet restore -s http://172.17.0.1:8080/v3/index.json -s https://api.nuget.org/v3/index.json
Overall, the solution works very well. On builds where the csproj files don’t change, I’m able to cut out close to a minute of package restores and when you scale that to hundreds or thousands of builds a day, that’s quite a savings.
If you are interested in full instructions for configuring your build/containers, check out the ThinNugetProxy repo.
I hope this helps you improve your hosted build times and please let me know any feedback you have!