Reducing compilation times

Compilation times are a big productivity killer and although D is considered to have fast compilation times, it still takes time. Also, when you go about your projects without any care for this sort of issue, it will eventually haunt you. At Tripaneer, we were — and still are — facing this situation.

Since there’s not a lot of information about it — there’s a ton of information about C++ but not much about D — and I also struggled a bit to understand what were the bigger factors contributing to increase compilation times, I decided to share our findings.

Use Gold linker

By compilation times I’m referring both to the compilation and linking times. However, if we separate those two phases, what was actually taking the most time in our case were the linking times. Fortunately, there’s gold linker which is a blessing. In our case it basically halved the whole compilation time: full builds went from around 1m and 50s to 1m and 3s and incremental builds went from around 1min to 27s. These are incredible gains.

In order to use the gold linker, at least in ubuntu 17.10, you just need to point the symlink /usr/bin/ld to /usr/bin/x86_64-linux-gnu-ld.gold like this:

sudo ln -s /usr/bin/x86_64-linux-gnu-ld.gold  /usr/bin/ld

There’s however bad news for MacOS users: the gold linker is not available for you guys :(. Since this option is not available for everyone, I’m going to continue using the regular linker in the rest of the article unless I mention otherwise.

Avoid ctRegexes

ctRegexes (compile time regexes) are great if you really need big performance and don’t mind the compilation time hit. For us, a regular regex is still good enough, I mean, in web development the bottleneck is rarely the code itself. Most likely, the bottleneck will be IO like a slow SQL query for example. So the first easy thing to do is to get rid of ctRegexes whenever possible. This was the first thing we did already some time ago and unfortunately I don’t have the results anymore. For some reason, I’m also not able to build the version before and after the ctRegexes were removed :(. However, I remember the results were substantial. Of course, it all depends on the number of ctRegexes you remove.. in our case, we removed a lot of them.

Make use of local imports

Something else we were doing a lot was to do all our imports at the top of each file even when you simply needed a function from an import in one very particular function. I went through the trouble of removing some of our global imports from the top and added it as locally and as specific as possible and was able to get 3 seconds in a full build.

So, instead of doing:

import bla;

at the top of a module making the import available in the whole file, I did:

void myVeryImportantFunction() {
   import bla : myImportedFunction;
   auto result = myImportedFunction();
   ...
}

It’s important to mention that so far I had only done it for our easiest imports, where I didn’t need to go and make the imports in a lot of places and I already got 3 seconds. Another important thing to mention is that the size of the file you’re importing seems to be really important as well. To get those 3 seconds I changed the imports for several small modules. Then I picked one really big module and went through the trouble of changing all the imports of this big module to make them local and managed to get another 6 seconds. So with only one big module the gains were twice what we got with several small modules.

Conditional compilation

All these tricks are nice but you know what really reduces compilation times? Compiling less code :) This is where D’s conditional compilation comes in handy. In our case we have several big pages so what I did was to create several versions for it in our dub.json (kind of the make file for D). Imagine, if I want to include page1 in the compilation, I would create the version page1, if I want page2, I would create version page2 and so on. Something like this in dub.json:

"buildTypes": {
        "dev": {
            "versions": ["development", "page1"]
        },
        "live": {
            "versions": ["live"]
        },

In this previous example, if I ask to compile dev I would get only page1 compiled. In order to compile build type dev only, I just need to call dub like this:

dub build --nodeps -b dev

The nodeps param tells dub to not check for dependency updates in the dub repository (which also helps to keep the compilation times lower)and the -b dev tells dub to build the build type dev which has version page1 enabled. Then, in the code, we simply need to do:

version(page1) {
// all the code I want to compile
}

This is where we get the biggest gains in compilation times since we reduce a lot the amount of code we need to compile. This allied to the local imports becomes extra powerful, since you also just import the strictly necessary modules.

Also, please notice that the live build type doesn’t contain any version. That’s because in live we always want to build the full project. It’s just in development that we want to build just what we’re working on. However, the previous code snippet doesn’t check if you’re in development or live version. For that we would need something a bit more elaborate:

version(development) {
  version(page1) {
    enum compilePage1 = true;
  }
  else {
    enum compilePage1 = false;
  }
}
else { // not in development version
  enum compilePage1 = true; // if we're not in development version
                            // always compile page1
}
static if (compilePage1) {
  // all the code I wanna compile goes here
}

Unfortunately you cannot do something like:

version(page1 || !development) {
  // all the code I wanna compile
}

So you need to make use of static if together with version .

Problems with conditional compilation

Be careful with conditional compilation though. Imagine you have module A which is used both in page1 and page2 and you’re simply building page1. Now you need to change module A and it becomes incompatible with the previous version (i.e: you change a function’s signature that is also used in page2). Since you’re only building page1 you will get no errors in page2. Well, with great power comes great responsability :)

Conclusion

So far, like I mentioned in the beginning, we have been a bit careless with compilation times, so it just kept increasing. A full build on my laptop was taking around 1m50s before any tweak. With all the tweaks regarding local imports and conditional compilation (we removed the ctRegexes already some time ago) we managed to reduce the full build time for one of our main pages to 1m24s (this means we’re just compiling that one page we are working on). These are full builds which we don’t really need to do that often. Normally, with incremental builds, it was taking around 1 minute for the full project and now takes around 34s (again, just one page).

When using the gold linker these gains are maybe a bit irrelevant as the biggest time, at least for us, is spent on the linking phase.

There’s still a lot to do but the cool part is that we are now getting to know some tools to get these pesky compilation times under control while we develop. Not using the gold linker, the total compilation time is still largely the same but at least we’re much more productive while developing since we can opt to just compile that page we’re working on.

Please let me know if you find some other worthy tricks :)