Photo by Ana Frantz on Unsplash

What is wrong with Gradle?

First of all, don’t get me wrong. I love Gradle, I love Kotlin and I love them together. In my opinion, Gradle is the best build system ever and I always encourage its use not only for Java and Kotlin but for other languages as well.

Still, the system is large enough now and starts to suffer from design decisions made, when it was much smaller. The problem I am going to talk about is not new, but it became apparent with the arrival of gradle.kts syntax.

How does a plugin work?

The main feature of Gradle is its modularity. One could do almost anything with Gradle if they have an appropriate plugin. One can easily take the plugin from a repository or write it in place. It is actually quite easy.

Unlike systems like Make, Gradle has a project model — a declarative description of what is there in the project, dependencies, and some metadata. Gradle utilizes convention over configuration doctrine, which states that if nothing is configured, the project model assumes reasonable defaults. Also, Gradle has tasks — actions to be performed based on the project model. Each task also has a number of typed parameters. Tasks could also depend on other tasks (both in the current project and in other projects), thus producing task graphs. One must note that so far the system is fully declarative. Tasks themselves contain the imperative code (and it is one of the main benefits of Gradle), but the whole configuration is written in a formal model during the configuration stage.

Now comes that plugin.

The Plugin interface is very straight-forward (let’s look into the official documentation):

class GreetingPlugin : Plugin<Project> {
    override fun apply(project: Project) {
        project.task("hello") {
            doLast {
                println("Hello from the GreetingPlugin")
            }
        }
    }
}

// Apply the plugin
apply<GreetingPlugin>()

The authors were obviously inspired by the famous KISS principle. The class has only one method, that applies the change to the project model. Then all you need is to instantiate the plugin class (Gradle does it under the hood) and apply it. It is simple. It is beautiful. Isn’t it. Does anybody see the problem yet?

Several actually.

Problem 1: not declarative anymore

Anyone, who worked with data systems should see the problem right away. There is something that changes the state of the configuration and there is a time of this change. It is bad. Before, there was no time in our system so there was no order of changes. Task graph does not count because it is formal and tasks are processed after configuration is finished. It is not the case for plugins.

The order in which you apply plugins affects the results. Also once you introduce time and the state, you can’t get it back. One of the plugins could rely on the stat provided by other plugins. Modules (in Gradle they are called projects) complicate things further because different plugins are applied to different projects and one plugin could affect tasks in different projects. The resolution order of module plugins is a grey area, there are not a lot of people, who understand it.

Another problem is that all parameters in the Gradle model are eager by default. Meaning they happen when you apply plugin and could not be ordered. The Gradle team is introducing new APIs to cover that in the latest releases, but it looks ugly. Like this:

Here we are applying everything inside outer lambda as a callback on the application of plugin with appropriate ID. Extensions are configuration objects created by the plugin, so they do not exist “from the start”. They could be resolved only after a specific plugin is applied. So one needs to first check if the corresponding plugin is applied (or react to its application via lazy initialization).

In the example above, I use a little bit more complicated scheme. I prefer the already applied plugin to the one I apply myself. It was done to allow providing a custom version of the Kotlin plugin, but it does not work that well.

This lazy configuration solves some problems with plugin application order, but not everything. Plugins are just not made to be able to configure other plugins. Each plugin makes additions to the project model, and seldom some changes to the existing model. But they do not have the means to communicate with each other. The typical case is the interference of Kotlin and java plugins. Kotlin relies on configuration (source sets for example) from the java plugin, but some plugins do not know a thing about Kotlin and try to use the default Java configuration (e. g. default main source set). In Kotlin-multiplatform, there is even a special withJava() marker that makes build expose source sets in a way the java plugin likes (still there is an issue with it as well).

Problem 2: tuning prohibited

The Plugin interface above does not have any parameters. Neither plugin class itself, nor the application method. Indeed, Gradle developers assumed that the configuration could be done via task parameters. And plugins were bare-bones in order to be able to use reflection for class instantiation via reflection.

A plugin can create an “extension” object available to the user during the configuration stage. And tasks could rely on the information in this extension. But extension could not regulate which tasks are created and how they depend on each other. There is a widely used hack, to pass the project model as a parameter to an extension like it is done here. It allows to dynamically affect project model from user configuration, but it creates even more confusion with the order of property changes.

On one hand, it seems to be a reasonable assumption. On the other hand… no, does not work. Take Kotlin for example (again). Kotlin compiler is embedded in the Kotlin plugin. So if you want to use a different version of the compiler, you need a different version of a plugin. But it seldom happens, that you need to modify the compiler version without changing the plugin version. For example, you need to use a custom compiler or just a different version of the compiler for testing purposes. Right now, one needs to rewrite the whole configuration when the compiler version changes.

Another problem with the API is that one plugin can’t depend on another plugin. One could, of course, do something like this:

But what about tuning? What happens that two different plugins depend on a single plugin and apply different changes to the configuration?

Problem 3: where does my classpath lead?

One needs to remember that the Gradle build is actually a regular JVM program. It is the greatest benefit and one of the greatest problems at the same time. Each JVM program has a classpath. All classes should be loaded in a tree-like structure and it is not possible (by simple means) to have two different versions of the same class. JVM allows loading classes on the flight, which is quite convenient for plugin-oriented systems, but it is in general not possible to “unload” the class (actually you can do that, but it is a long discussion). And it is definitely not possible (without introducing a lot of clutter) to load two versions simultaneously.

Now, we remember that plugins do not have parameters. So the only way to implement different behavior of a plugin is to make a different version of the plugin. And here we have a problem with classpath. If one module needs one plugin version and another module needs another plugin version, we can’t put them in the same project.

Furthermore, the build system classpath must be resolved before the official configuration stage. Meaning that we have three actual stages:

• Buildscript configuration
• Project configuration
• Task evaluation

So there are three different classpaths — one for plugins, one for project model after plugins are applied, and one for the actual build. It is hard to understand which version goes where. For example, Gradle has an embedded Kotlin compiler, but its version is usually different from the one used in the project. One can explicitly set version in the plugin application block, but it is not always straightforward, because it will fail if the plugin artifact is already on the build classpath (for example it is required for another plugin). So we have two different layers of plugin dependencies on each other — classpath dependency and apply dependency.

You probably noticed that parts of the script like buildscript block or plugins do not seem to be in the same “space” as the rest of the build script. Indeed, those blocks are evaluated at a different “time”.

Plugin management block in Gradle source code

The snippet above shows that the block is not even evaluated. Those blocks are captured by Gradle compiler hacks and interpreted before the rest of the code to set up classpath. This brings us to another problem.

Problem 4: compiler hacks

One of the examples is listed above. Another, much more obvious example is the plugin block in the main build.gradle.kts file. It works the same way. It is executed before the build script, so it can’t access the body of the script.

Here, Kotlin scripting is playing its role. In case you did not know, Gradle uses a kind of staged compilation to provide Kotlin autocompletion. It takes loaded plugins, then creates a set of synthetic extensions which are implicitly imported into the build script. On one hand, it provides a seamless autocomplete and static access experience for gradle.kts users (and it is really nice. On the other hand, in order for this thing to work, one must know plugins in advance. And if you want to declare plugins in the same file… well… you need magic. Magic is not bad itself, but in programming, magic usually produces inconsistencies. And inconsistencies are bad.

For example, one usually wants to use a version number both in plugins block and in dependencies block. But it is not possible. Plugins block lives in its own “time bubble” and can access outside code. So one writes perfectly sensible Kotlin/Groovy code and it just does not work like intended.

Solutions?

In the beginning, I said, that I like Gradle a lot. With all its flaws. Still, those problems must be addressed for the tool to progress. Or it will be replaced by something more consistent.

I think that several things should be done to solve the plugin problem:

• The system should formally introduce an additional zero build stage — build management stage (before configuration and execution stages). This stage must be configured separately, not inside build.gradle.kts . Maybe it should be done in settings.gradle.kts, maybe the other way, but not as a hack in the build file. In Kotlin it could be done via @file: annotations. But maybe there are other better ways. Plugin versions must be declared outside of the build file. There is an effort to do that in Gradle via version catalogs, but so far it is tricky to make it work (see here for example) mostly due to problems 3 and 4. There are additional compiler hacks (now in settings.gradle.kts) and classpath loading quests.
• There should be a new type of plugin (super-plugin? feature?), that has a parametric factory and could depend on other factories. Those super plugins should be able to declare configuration extensions without application (before application). And application itself should be done with minimal side effects. Doing this would allow setting explicit plugin dependencies and parameters for plugins. Also, detect parameters conflicts during plugin dependency resolution.
• I think there should be some kind of mechanics for layering configurations (similar to how it is done in https://github.com/mipt-npm/dataforge-core, see the article for explanation). This would allow avoiding conflicting configuration changes from different plugins.

All those things do not require creating a brand new system, they could be done in Gradle on top of the existing project model. Some steps are being done in this direction. But most of them address symptoms (configuration order, version catalogs, etc), but not the problem itself — plugin management model. So I think that there should be an effort to design how the thing should work and then try to put parts of it into the system, parallel to existing features, not trying to replace them. It is a huge work to design those changes and test their compatibility with existing systems. But it should be done. Either in Gradle or in new systems that will come after it.

I would like to thank Vladimir Sitnikov for reading the article and leaving a lot of comments. We both like Gradle, but in some cases, our opinions are quite different, so I did not agree with all remarks. And still those remarks made the article better.