From .NET Framework to .NET Core: History of a C# build farm
As we mentioned in the previous posts of this series on .NET Core, Criteo started as a full .NET shop.
To understand the challenges we faced when starting on .NET Core, we will go back to the 2010s and take a look at the .NET build farm we created to manage our codebase. When MSBuild 15 and Visual Studio 2017 came out, this build farm, that worked reasonably well until then, became our main challenge in successfully transitioning to .NET Core.
This article will attempt to paint a brief history of build at Criteo, with the evolutions of MSBuild that went with it.
Our build pipeline
Humble beginnings
In 2005, our codebase started as any other codebase: MSBuild .NET Framework projects in a single repository. What works with a few projects becomes unwieldy as the size and the number of projects grow:
- In 2011, development branches were integrated weekly and took 90 minutes to build.
- In June 2012, the same process took 6 workdays: A weekly integration was no longer possible.
That’s when we realized that something needed to be done: The codebase was separated in multiple repositories built independently from the others and NuGets were used to publish artifacts.
But it was not long before it led to another kind of nightmare: Dependency hell. How to ensure that the versions of an assembly (i.e. the binary result of a built project) were consistent through all the repositories it was being used in? And for all assemblies!
During 2013, the issue was so huge that we reached the point where no new feature was released for 3 months.
Build from source
At that time, an internal project called “red pill” was started to improve our release process. On the build side, it introduced a new principle for C# development: All internal code should always be built from source in a single pipeline and with the same version. Only one version (and by the way, no more versioning was used) of an assembly was used in all projects.
But to implement that principle at the scale of Criteo required some tooling on top of MSBuild. Enter a new command-line tool: CBS or Criteo Build System.
The main goal of this CLI tool is to automate cloning and building of all repositories as a single entity inside a folder called a workspace.
But this tool also allows a developer to create workspaces with only a subset of repositories checked out. CBS in that case uses artifacts produced by the CI pipeline to avoid building the missing dependencies locally. We call this mode of operation “partial checkout”.
How is partial checkout implemented?
MSBuild implements references between assemblies in a way that needs a bit of explanation before going any further.
It all begins simply enough: MSBuild introduces the notion of “Items” that describe the inputs of the build system. The .NET SDK uses items of type “Reference” to track the references between a project and its dependent .NET assemblies.
It also uses items of type “ProjectReference” to describe references between projects. A very crude way to describe a ProjectReference is that it will be transformed into a Reference to the assembly built by another project.
This is where CBS implements partial checkout: References between projects are declared as simple references. It can then adjust where to look for a dependency assembly whether the repository for that particular assembly is available locally for build or not.
In a given workspace, CBS knows which project sources are checked out and so it can handle the build order itself when running from the command line. CBS can also create a Visual Studio Solution with proper build dependencies for the developers to use in their IDE.
How about external dependencies?
Before Visual Studio 2017 and MSBuild 15, external dependencies were handled in .NET with a separate tool called NuGet. Its role is to keep a list of external packages required by a project in a file called “packages.config”. When a package is added, the tool will also patch the content of the project to add the necessary Reference declarations, including the location where the assembly was downloaded by NuGet.
CBS detects the references created by the NuGet CLI and automatically installs the associated NuGet packages in the workspace.
How about transitive dependencies?
We explained previously that we replaced ProjectReference by Reference to be able to implement partial checkout. But this is the root cause of two problems.
First, MSBuild no longer adds transitive dependencies to the C# compiler command line. This means that a given project must add references to all its dependencies required for compilation.
Second, when MSBuild constructs the list of assemblies to collect in the output directory for runtime, it no longer has a global vision of the different projects involved in a build. It only knows about the produced assemblies for the current project.
For example, let’s consider Project A, which has a reference to Project B, itself having a dependency on NuGet C. Since A has no knowledge of C, how could it know where to look for it?
ResolveAssemblyReference task
Enters “one of the most important tasks in the MSBuild toolset”: ResolveAssemblyReference (RAR). It is also the source of many headaches in the .NET community because of its complex job.
Its goal is to analyze all known references of a project and match them to file locations on disk. It will do so recursively by analyzing the metadata of each assembly to discover its own set of references.
CBS uses this task to give MSBuild sufficient information about the dependencies of projects from a repository missing on the local workspace. CBS gets this information from an “AssemblySet”: Built in the CI pipeline, it is a complete graph of all assemblies, references, and NuGets for all repositories.
When building on the developer’s machine, this AssemblySet information will be used to give the RAR task extra disk locations to look for missing assemblies.
2014 victory
After switching to the new system, our .NET build was back on track. A second version implementing a build cache of assemblies allowed us to further improve the build time. That cache uses git repository commit hashes to determine if a particular assembly needs to be rebuilt or simply downloaded.
The previous diagram summarizes the performance of our CI. Splitting our repository to 130 repositories/pipelines allowed us to temporarily keep up on releasing every week in 2012. In 2014, the introduction of “build from source” reduced the integration time to 40 minutes. Finally, the introduction of the caching mechanism allowed us to keep the performance while doubling the number of repositories.
All was well until MSBuild 15 came along…
The beginning of .NET Core at Criteo
dotnet CLI and project.json
When we started working on .NET Core in 2017, the Microsoft SDK was using a new build system based on JSON files. To be able to run our first tests on .NET Core without redesigning completely our tooling, we started simple: We created a separate pipeline that would clone all sources and simply call the dotnet CLI to build what we had already ported.
Because there were very few projects ported to .NET Core at the beginning, this proved sufficient to begin our first experiments.
Updating our external dependencies
The second step was to start working on our external dependencies to make sure that they were compatible with .NET Core. We already had a tool to check how far behind we were on the latest available NuGets so we simply improved it to include information about .NET Standard compatibility:
Visual Studio 2017
With the release of Visual Studio 2017 and MSBuild 15, .NET Core officially deprecated project.json to go back to MSBuild as the build system for the .NET Core SDK. This helped us with the migration, but the new .NET SDK projects proved challenging to integrate in our custom design for CBS.
The changes in MSBuild
To support .NET Core and the principles it introduced, MSBuild was modified quite significantly. A new notion of SDK was introduced to allow automatic import of tasks and targets into a project. The primary SDK introduced, called “Microsoft.Net.Sdk”, was designed to build assemblies targeting .NET Standard, .NET Core, and .NET Framework.
But projects that used the new SDK have a structure quite different from the old ones:
- They rely a lot more on convention rather than configuration: the list of source files is gone, almost all properties have sane default values, some items metadata are simplified.
- A new type of item is introduced: PackageReference. It brings tight integration to NuGet inside MSBuild. It also means that a new target is introduced: “Restore”. Its job is to solve the package dependency graph and download the needed NuGet packages.
These changes required modifying CBS in order to support them. As CBS was directly parsing .csproj files as raw XML, the first thing that we had to do was to improve the parser to support both formats. We also had to include the new Restore target in the sequence of targets that CBS would call during the build.
At the same time, we also started to work on a tool that could automatically convert old projects into new ones.
.NET Standard
.NET Standard is the definition of a set of APIs that are common to the implementations of both .NET Framework and .NET Core.
As there is no true .NET Standard library (the implementation of these APIs is different in both frameworks), a bit of magic happens. When a project targets .NET Standard, MSBuild needs to provide assemblies to the C# compiler so that it can properly compile the code and only use the shared API.
To do so, the SDK contains a set of assemblies (called references assemblies) used for compilation only. They are simply classes and methods with empty implementation.
At runtime, when the final framework is known, these facade assemblies are replaced by implementations that contain type forwarding declarations, so that the actual type used is the one from the proper framework.
The SDK that shipped with MSBuild and .NET Core contains extra code to deal with this specific issue. But it effectively broke our approach in handling transitive dependencies: as most of.NET Standard dependencies are shipped as NuGet packages, they are treated as such by our tooling that has no notion of these new types of assemblies and how to use them to create an appropriate search path for the RAR task. We ended up with DLLs incorrectly selected for the final target framework: for example, reference assemblies sometimes appeared in the final runtime folder, instead of their true implementations.
Double build
Our first impression was that we could use .NET Standard to progressively convert all our projects to the new SDK and .NET Standard. But due to the implementation details of both our tooling and the magic detailed above, it would not have been possible without a major redesign of our CI system, which we didn’t have time for.
The solution we ended up with was to create extra MSBuild projects, next to our existing .NET Framework projects, with a “.netcore” suffix.
These projects simply reference the sources from the other folder but are completely separate .NET Standard/Core projects, that produce an assembly with a different name.
This allowed us to get developers started and make sure that their code was compatible with .NET Standard. By the way, this is still how we run our build today.
What we are working on
Pending issues
Though we have successfully created a build farm that can support all our .NET needs, it still has a few limitations.
The first obvious problem with our current approach is the duplication of projects. Having 2 projects means 2 builds, twice the build time, and poor user experience in the Visual Studio IDE. Each new dependency must be added to 2 projects. This will probably disappear as we move away from .NET Framework, but it may take a long time to get there.
The second issue is related to the way we handle dependencies. CBS has gone too far from the MSBuild system and requires us to duplicate parts of MSBuild logic in our tooling. The main issue we are facing is that we incorrectly propagate dependencies to client projects. The developers have to add extra dependencies that are not strictly needed at their level but required by some other NuGet/assembly they are depending on. It also means that, with each new release of MSBuild, significant work needs to be done to issue a new version of CBS.
The last problem we have to struggle with is consistency: Because we provide MSBuild with only a partial view of the dependency graph, all the safeties around package version conflicts and incompatibilities are effectively not enforced and would be too painful to reimplement and maintain in our tooling. These problems usually show when an incorrect version of a particular assembly is selected by the RAR task, but not properly selected at runtime.
Our next move
We have created a new version of our build system, completely MSBuild-based. It relies on the new .NET SDK type of project, and implements partial checkout in the following way:
- References to internal projects that are present locally are replaced by ProjectReferences early in the build.
- References to missing internal projects are replaced with ProjectReferences to “placeholder” projects. Generated from the AssemblySet, they contain only Package and Project references, and point to the downloaded artifact from the CI. The long-term goal is to replace these placeholder projects with real NuGets and PackageReferences.
However, during the experiments we ran on our current projects, we realized that our dependency graph was full of inconsistencies that prevented us from using it without fixing them first.
Because of the size of our codebase (250 repositories and 3000 projects), this is still an ongoing effort. Our tooling may also need more work to keep our build performance at the same level it has today.
Conclusion
This article was a very brief overview of the build system for .NET at Criteo. It allows us to implement our company-wide “build from source” strategy while keeping decent CI times given the size of our codebase. The design choices made at the origin proved challenging to adapt to new MSBuild versions.
As a result, our transition to .NET Core and MSBuild 15 Is still ongoing but allowed us to successfully build and release our first products to a Linux production platform.
With the latest additions to MSBuild, we hope to have a design much closer to the .NET SDK, so that later versions of MSBuild will be hopefully much smoother to integrate.
Want to read more about the topic? Check out the series:
Join the journey!
