Yandex Publishes Yatagan, an Open Source DI Framework for Quicker App Builds

Published in

Yandex

28 min readDec 14, 2022

My name’s Fyodor Ignatkevich, and I work on the Yandex App and Yandex Browser for Android. About a year back, I suggested the idea of a dependency injection framework to the team. The framework ended up almost halving the build time of our projects. We’ve published the framework to GitHub so that other developers can speed up their builds as well. I’d written the framework from scratch; then, together with the team, we integrated it into our projects and actively use it to this day.

My development experience is precisely what I want to tell you about. Let’s try to figure out what are the factors that slow down the build when using Dagger, and how Yatagan, which is compatible with Dagger in terms of API, copes with them. We will also consider what other tasks the DI framework may face — for example, native support for dependencies under runtime conditions, which is present in Yatagan and which spared us from manually processing the states of A/B experiments in DI.

The article covers many technical nuances that I discovered while researching. In the end, we’ll see what projects can benefit the most from Yatagan as a substitute for Dagger and what projects wouldn’t benefit much from it.

The Project

To fully understand the technical decisions I made while developing Yatagan, we need to start with a brief look at the project it was initially designed for: ~150 Gradle modules, ~2 million LoC in Java/Kotlin, with rich history and quite a bit of legacy code and solutions. What matters for us here — it’s big. And that may make build times a problem.

Products and Experiments

Our project’s codebase can be used to build multiple apps, each with their own features. It also extensively uses A/B experiments to test hypotheses and evaluate how code changes impact key app metrics.

In code we need to process experiment states, so let’s introduce the following structural model: sections of code can be turned on or off depending on conditions defined during the build process (static) or evaluated at runtime (dynamic).

Static conditions stem from the fact that we can build multiple apps and need to modify the behavior of each one.
Dynamic conditions are defined by the state of A/B experiments in the client at a particular moment in time.

Such conditions can impact anything: from slight alterations made to algorithm parameters to a whole new features for the user. They might be localized or appear throughout the code.

For DI (dependency injection) written in Java/Kotlin, this model is implemented by treating conditionally engaged sections of code as classes plugged into DI graphs. The presence or absence of a certain class in the DI graph is determined by a corresponding flag — whether an experiment is enabled (dynamic condition). Static conditions express whether a class is present in a DI graph for the specific app.

From Reflection to Code Generation: a Brief History of DI in the Project

To understand how and why we settled on Dagger, let’s take a quick tour into the history of DI in the project. Dagger has a complicated API which gives you a number of different options for setting up DI. I think this does more harm than good: experience has shown how difficult it can be to properly set up DI using Dagger, and rewriting such code is both expensive and painful for large projects.

Reflection — IoContainer

We’ve been working on this project for a while (since 2013), starting in a pre-Dagger world where there already was Square/Dagger but not yet Google/Dagger. Square’s first reflection-based version wasn’t very popular, and our team leads decided we should write our own simple DI framework: IoContainer. It’s your traditional service locator that requires explicit registration of all classes in the DI.

Sample class:

public class MyImpl implements MyApi {
    @Inject public MyImpl(
            Activity context,  // Standard (direct) dependency
            Optional<FeatureSpecific> optionalDetail  // Optional dependency 
    ) {}
}

Sample registration at the start of an Android app:

registar.register(Activity.class, activityInstance);  // Provided class instance
registar.register(MyApi.class, MyImpl.class); // Interface registration, aka @Binds
if (myFeatureA || myFeatureB) {
    // Conditional registration
    registar.register(FeatureSpecific.class);
}
IoContainer.complete(registar);  // Finishes registration

And usage:

IoContainer.resolve(context, MyApi.class)  // Yields MyImpl
IoContainer.resolveOptional(context, AnyUnregisteredClass.class)  // Yields Optional.empty()

The dependency graph in IoContainer is inherently dynamic, and optional dependencies are pretty straightforward to do: if no registration is made — class is not present. The absence of code generation makes for a quick build, too. Implementation is a simple hierarchy of dictionaries with some helper code around to serve it. It is very simple, yet I have to say, years of use and improvement had made it a highly performant tool at that.

But there is one major drawback: there’s no validation to make sure the graph is valid for all combinations of conditions. If a class requests a dependency that wasn’t added to the container, a MissingDependencyException is thrown. Autotests and QA team can obviously only test a few configurations out of a combinatorially large number of possible ones. The alpha version had a habit of crashing because of that, and that wasn’t plausible at all.

2. Dagger: the Beginning

Dagger 2 started gaining popularity, and the team began to consider migrating. It would solve IoContainer’s biggest weakness — MissingDependencyException — since the whole dependency graph is verified while the project is built. We also figured this would make the app start faster.

Unfortunately, Dagger doesn’t have any native support for dynamic graphs. While static conditions can be expressed with @BindsOptionalOf (we ended up not using that anyway), dynamic conditions are far more complicated to express well in Dagger. Using the same terms as the IoC example above, this was the most concise we could make it:

@Provides 
Optional<FeatureSpecific> optionalOfFeatureSpecific(Provider<FeatureSpecific> provider) {
    if (Features.myFeatureA.isEnabled() || Features.myFeatureB.isEnabled()) {
        return Optional.of(provider.get());
    }
    return Optional.empty();
}

There should also be a binding to Optional<Lazy<FeatureSpecific>> if anyone needs a Lazy option.

When written like this, there was nothing stopping a developer from accidentally depending on a FeatureSpecific class directly and using it even when the conditions weren’t met. And this is just one issue; the other ways Dagger modules could be set up for simulating conditions had their own pitfalls, including the fact that none of them were particularly compact in terms of code.

Looking back: How I Think the Optional Functionality Problem Should Be Solved Using Dagger
There is a good solution for setting up an architecture to meet that need, of course. Extract an interface for the optional functionality in question and write “stub” implementation in addition to the real one. And within @Provides method you can return either real or stub version depending on the condition. This way, external code doesn’t have to handle the fact that dependency class can be present or absent, so you don’t have to use Optional<T>.
Here’s what then replaces the code in the example above:

@Provides MyApi provideMyApi(Provider<FeatureSpecificApiImpl> implProvider) {
    if (...) return implProvider.get() else return new MyApiStub();
}

For projects that don’t use too many conditions in their DI, this is a great option. Issues only start to arise when you try to split the interfaces for each condition and design the API for them to be able to provide a no-op implementation that matches the interface contracts.
When it comes to our project, which became riddled with conditions all over, setting up API facades and stubs would have meant major refactoring. IoContainer came back to bite to us: using Optional<T> had been free, and our code was full of optional dependencies.

The further we progressed in migrating the project to Dagger, the harder it became to support the DI code because of all the conditional bindings we had to write. There was too much room for error, and the DI code related to conditions was very hard to navigate and read.

3. Dagger + Whetstone

As soon as we started migrating to Dagger, we saw the problems that conditions posed. That’s when I decided to make a companion framework for Dagger that could (among other things) take our runtime conditions and generate Dagger modules for their handling with a code that we would write ourselves. And it had to be safe. I called this thing “Whetstone” (don’t confuse it with the same-named open-source tools, ours wasn’t published anywhere).

The key trait of Whetstone was the special Condition API that let us declare dynamic conditions right on top of bindings and classes. First, we had to declare a “feature” — a special construct annotated with @Condition. For example:

@AnyCondition(
   // Means: find static method/field MY_FEATURE in Features class,
   // find method/field isEnabled for that value, the result Boolean.
   // Rejections are indicated with '!' at the beginning of the line.
   Condition(Features::class, "MY_FEATURE_A.isEnabled"),
   Condition(Features::class, "MY_FEATURE_B.isEnabled"),
)
annotation class FeatureAorB

@AllConditions(
   Condition(Features::class, "MY_FEATURE_A.isEnabled"),
   Condition(Features::class, "MY_FEATURE_B.isEnabled"),
)
annotation class FeatureAandB

These conditions let us encode any Boolean function in conjunctive normal form. @Condition declares a Boolean variable with optional negation. Applying the AND operator is then a matter of annotating the feature with multiple @Condition annotations (for Java) or applying @AllConditions (for Kotlin). If we needed the OR operator, we went with @AnyCondition. Here’s how that feature could have been used:

@BindIn(SomeModule::class, condition = FeatureAorB::class)
class UnderAorB @Inject constructor(/*...*/)

@BindIn(SomeModule::class, condition = FeatureAandB::class)
class UnderAandB @Inject constructor(/*...*/)

// Where SomeModule looked like this:
@Module(includes = [WhetstoneSomeModule::class])  // This is the module that is generated by Whetstone
interface SomeModule { /* */ }

Afterwards, Whetstone collects all @BindIn from the project and groups them by target module. It then generates “companion” modules with all the boilerplate code.

Whetstone could do a few other tricks:

Guarantee that each unique @Condition would only be evaluated once (special holders are generated to cache conditions).
Process constructs like this:

@BindsAlternatives
fun binds(
  a: FeatureSpecificImpl1,
  b: FeatureSpecificImpl2,
  c: DefaultImpl,
): Api

This binds the first available class in the graph to the interface. In other words, if there was FeatureSpecificImpl1, the Api query would return it. If not, it would return FeatureSpecificImpl2. If neither condition was met, it would return DefaultImpl. If even the last alternative is under condition, the whole Api object would be conditional.

Perform a few other things specific to our project’s DI that were easy to automate, like doing automated subscriptions to events. These, however, didn’t make it into Yatagan, so we’re not going to discuss them in detail.

Still, the most important thing Whetstone gave us was a compile-time guarantee that dependencies between conditional classes were valid in every possible case. In other words, classes couldn’t depend on each other directly based on an incompatible condition. The framework had special condition validation code written into it. Verifying binding A’s dependency on binding B came down to proving the statement: “if the condition for binding A is met, then the condition for binding B is met for all input condition values”.

For Those of You Who Prefer a More Formal Language

Then verifying the direct dependence between binding A and binding B comes down to verifying the truth of the following Boolean expression:

That’s a task for a nSAT solver.

Theoretically, that’s an NP-complete problem for a Boolean satisfiability solver. Whetstone used a fairly straightforward DPLL algorithm implementation. In practice, such condition validation didn’t incur any performance problems as the size of input expression was relatively small.

Sample Condition Dependency
Let’s look at the classes from the example above:
The UnderAandB class could depend on the UnderAorB class, because if the expression A && B is correct, then the expression A || B is also correct. In those cases, we can write a direct dependence: class UnderAandB @Inject constructor(ab: UnderAorB, ..).
The opposite is not true: the UnderAorB class can’t depend directly on UnderAandB since the truth of A || B does not follow the truth of A && B. Here we have to write an optional dependency: class UnderAorB @Inject constructor(ab: Optional<UnderAandB>, ..).

This migration eventually resulted in a hybrid DI framework that didn’t allow us to make mistakes in our dynamic conditions, generated code, and gave us a faster app startup speed in release configurations over reflection. But there was a major drawback: project build time increased significantly. Let’s take a look at Dagger (+ Whetstone) from a technical point of view to understand what slowed the build down and how it can be fixed.

“Slowly but surely”, or Problems With Build Speed

After some time with Dagger + Whetstone, we realized something was wrong with our build performance. Since we were migrating gradually, there was no sudden performance drop. That’s why it took us a while to realize there was even an issue. After numerous complaints about slow builds and a fair share of meetings, I got to work answering the question: exactly which technical aspects of our configuration are sabotaging build speeds?

We’re going to look through everything I noted in my research. We’ll start with the obvious.

Problem 1: kapt

If you drop by a mobile dev office that uses Dagger and ask any random developer there what makes Dagger slow down build times, they’re probably going to say: “kapt”. And they’d be right. kapt has to run Kotlin compilation in the special mode to generate stubs.

Stubs are Java sources generated from Kotlin without code in method bodies. They let traditional annotation processors (AP) like Dagger “see” code in Kotlin, but since it requires Kotlin compilation to be run (even though in the special mode — no method body compilation is done), stub generation takes a fair time to complete, which translates into a lot of extra waiting. This majorly affects cold builds and to a lesser extent incremental builds. So, the need to use kapt is one of Dagger’s biggest drawbacks. But let’s keep going a bit further.

Problem 2: Generated Factory Classes

Dagger generates a special factory (e.g. MyClass_Factory for MyClass) for every class with an @Inject constructor and every @Provides method. That’s a problem for number of reasons:

There are a lot of these classes and they bloat the amount of bytecode in the app (in the debug configuration without ProGuard/R8), slowing down app startup (app has to load more classes).
They also have to be compiled, which slows down the project build.
They often show up in search results and lists of classes in the IDE, which is annoying and makes it harder to navigate the project.

I should also mention that generating those factories forces the user to include Dagger in every project module that has at least one @Inject/@Provides. The project will build and even work correctly if Dagger isn’t applied to such modules, but there’s a catch for incremental builds. Gradle supports incremental annotation processing when specific conditions are met by the AP and the project that uses it. What’s important for us, is that the AP can only generate code for the program elements in the current compilation unit or Gradle module. If we tried to generate code for a class from the library (dependent subprojects are already libraries for the current one), Gradle would output a warning about missing originating elements, let us know that the incremental compilation can’t proceed and ultimately fall back to full recompilation. The problem arises if Dagger isn’t applied to the dependent module, which makes it try to generate factories for classes from that dependent module from the current one. And Gradle doesn’t like that. On the other hand, including Dagger in every module with at least one @Inject would almost certainly dramatically slow down the build with all the extra stubs and waiting for the AP. But ignoring the incremental processing system in Gradle would sacrifice incrementality.

And that’s not all. There’s a cherry on the top. Factory classes aren’t even required when Dagger is working in dagger.fastInit mode (more about that here), and that’s the recommended mode for big Android apps! Why does Dagger still generate them? Odds are it has something to do with binary compatibility: if someone wants to use them in dependent projects without fastInit, they’ll need those factories, and Google apparently decided to leave room for that use case despite how rare it may be. Then again, it’s only my guess, there may be other reasons.

Ultimately, factories just created lots of problems without really being needed.

Problem 3: Whetstone Implementation Quirks

Whetstone has two nearly independent parts: the generator and the validator. The generator is an AP that generates code for Dagger. The validator is a plugin for Dagger that uses Dagger SPI to verify the correctness of conditions for Dagger graphs.

The work of the generator part of the Whetstone is ultimately based on the assumption that Dagger would see the generated code.

For reference: How Are Dependencies Between Different APs Processed in the APT/kapt
In JSR-269 (the original annotation processing specification for Java), there’s no well defined order of calling multiple APs, and there’s nothing one can do about it. Instead, APs can work in multiple rounds. With each processing round the system (apt/kapt) invokes an AP, which tries to do its job. If there are unresolved elements in the code, it might give up and let the system know that the code is incomplete (some referenced classes are absent from the classpath). The system then marks the AP as pending and moves on to the remaining APs in the hopes that they’ll generate the code the first AP expects. As soon as all the APs have completed their work with some still pending, the system kicks off a new round and launches pending ones again. That continues until all processors have completed their work or a round passes with no new code showing up.

Because of the approach with multiple rounds, Dagger could be called before Whetstone and try to build its graphs. Then it would come across the nonexistent Whetstone~ modules and finish processing, handing control over to Whetstone. When Whetstone finishes processing, Dagger would parse the graphs again, duplicating the part of the job it had done at the beginning. This means that Dagger + Whetstone worked slower than they would have if they’d been a single processor. Moreover, Whetstone has to perform additional work to integrate well with Dagger and prevent developers from misusing its API and that didn’t reflect well on build performance.

One important thing to note is that Whetstone had to generate a significant amount of code for all optional bindings. It also “shadowed” the @Inject constructor for the classes under condition with a @Provides method to better control how it’s created and used. But that also added more bloat to the bytecode and further slowed build time and app performance in the debug configuration.

In Gradle terms for incremental annotation processing, Whetstone was an aggregating annotation processor. That means it had to collect syntactically independent constructs marked with @BindIn from all over the project and process them as a whole. When changes were made or a new class with @BindIn was added, the process had to be repeated in its entirety. That almost completely eliminated any benefit of incremental compilation given how many of those classes there were. If you’ve worked with Dagger Hilt, it functions the same way with @InstallIn: convenient, but slow.

Worth noting, that to write an “optional” binding in Dagger, as we’ve seen earlier, either manually or automatically, you have to depend on Provider<T> in order to decide whether to call provider.get() inside the binding code. However, looking at the code Dagger generates in different cases, I discovered an interesting detail. If a class is queried anywhere as Lazy/Provider, Dagger switches to a less optimal code generation strategy for it to be able to pass the provider object outside. Optional bindings were “spoiling” lots of classes in the graph since they were introducing “provider” usages to it. So this could theoretically affect generated code runtime performance.

Let’s recap the problems we found in Dagger + Whetstone for build speed:

It uses kapt, which is slow.
Dagger generates unnecessary factories, resulting in slower builds, more classes, the need to include Dagger in more modules, which requires even more time to generate stubs.
Whetstone slows down annotation processing, generates more classes, and forces Dagger to generate suboptimal code.

Yatagan, aka Dagger Lite

Once I’d gone through every item behind the project’s slow build times, I suggested writing “yet another DI” framework: a new engine for a Dagger-like API designed to support Whetstone’s main functionality for working with runtime conditions. I figured I could address all of the problems Dagger and Whetstone brought. I also decided to try natively supporting a Java reflection mode to speed the build up even more for specific cases.

I internally called the framework “Dagger Lite”, and while the team called it that too, I’ll use the public name from now on to avoid any confusion: Yatagan.

Here’s what I was going for and how I proposed we solve our issues:

Yatagan wasn’t going to generate any SomeClass_Factory or MyModule_Method_Factory, which would immediately solve a number of problems. That meant that it would be sufficient to apply Yatagan only in project modules that contained root @Component declarations, since the code is generated only for them.
Yatagan was going to natively support runtime conditions with a Whetstone-like API and generate optimal code for them. That fixed multi-round processing, redundant Providers and code bloat.
Yatagan was going to support the traditional kapt as well as the new KSP engine, which Google claimed was going to fix kapt’s performance problems.

I also suggested a few other optimization ideas:

Yatagan was going to natively support a reflection-only mode. This would be aimed for local development, where apps need peak rebuild speeds after modifications instead of peak performance. In the reflection mode, Yatagan would build the graph completely in runtime using information obtained via Java Reflection. That meant builds didn’t have to include annotation processing, making them significantly faster. For massive graphs, building them during execution can slow app startup by a few seconds, but that’s generally not as critical when compared to accelerating the build. Time from making a change in the code to actually seing the result in a launched app was going to be lower, so the developers could be more productive.
Yatagan was going to generate code for single- and multi-thread use as needed. The single-thread option features improved performance by stripping away synchronization and helper objects. The thread-safety option can be chosen separately for each graph hierarchy.

But there was one big compromise we had to make: it was only going to have a Dagger API subset.

There are parts of the Dagger API that almost nobody uses (like dagger.producers.*), so we shouldn’t worry about them.
Some parts, including dagger.hilt.*, were out of the question entirely due to Yatagan’s technical nuances (more on that later).

Some parts could be rolled out later as needed by us or the community.

Let’s get right to a few questions you might have about these suggestions. First, let’s take reflection-only mode. We already have Dagger Reflect, which does essentially the same, but we wouldn’t be able to use it since it doesn’t include Whetstone functionality. It also is a separate project, so there’s no guarantee that it will behave the same as Dagger with code generation. And that’s a guarantee we need to reassure developers that their code would behave identically with reflection and without it.

In terms of KSP support, Dagger is planning to make that happen on its own. There’s also Anvil. Dagger still doesn’t support KSP at the time of writing (and after the work I’ve done, I can understand why the work takes so long), Anvil — as a tool for accelerating Dagger — doesn’t give as much as Yatagan.

Why Is That?
Anvil is a plugin for the Kotlin compiler, which can generate all of those factories itself without resorting to kapt to keep the user from needing to include Dagger and generate stubs in extra modules, so it partially solves only one of the mentioned problems. Yatagan doesn’t generate them at all.
Concerning Hilt support
Dagger Hilt is positioned as a separate product and, in my opinion, is aimed at new small mobile development projects for Android. It was specifically developed to offset the high cost of initial DI setup for Android apps that came with classic Dagger. In contrast, Yatagan tries to solve problems for large projects already using Dagger. Most small apps don’t have the issues with build speeds that we do. They probably don’t need a native support for conditional bindings either.
Yatagan might be able to offer small projects something in the future, but that’s still down the pipeline.

The most important requirement was probably that the new framework had to be compatible with Dagger in terms of API or at least not require major migration from Dagger. The Yatagan API was based entirely on the Dagger2 API, completely replicating it in certain places. It absorbed Whetstone with a few changes: @BindIn was replaced by the @Conditional annotation. While Yatagan behaves differently than Dagger in some places, I did my best to document where. It uses @Condition/@Conditional to support dynamic conditions, and it replaces @BindsOptionalOf for static conditions with a system of variants that are like flavors/variants when building Android apps. We won’t get into API specifics in this article, but you can find them in the Yatagan API documentation.

Goes without saying that application performance is critical for our projects, so I did a lot of performance testing to ensure that the code generated by Yatagan is at least as good as the Dagger code in this matter. And in some cases it might come out faster as well. The code generation strategy is mostly based on Dagger’s “fastInit” mode.

I should say that Yatagan was never intended to completely replace Dagger. Even though it has the same core functionality, Dagger has some features that Yatagan is missing.

Yatagan: Architecture and Implementation

To implement all of the things I’ve mentioned, especially simultaneous support for three backends — kapt, KSP, and Reflection — I needed the right architecture. Yatagan had to behave exactly the same regardless of the backend. Let’s see how I did that.

Yatagan’s architecture is built on several basic abstraction layers:

1. :lang — An abstraction of the language model. It models types, classes, methods, and other programming language elements. For the most part, elements followed Java semantics rather than Kotlin since, from a type system point of view, Dagger works with Java types. I decided to stick with that for Yatagan. At some point, lang had initial native support for some entities from Kotlin (properties, for instance), but that had to be removed to maintain performance (keep reading). The API only contains what the next abstraction layers need as well as a minimum kit for plugin developers (see the docs on how to write plugins for Yatagan). lang has three main implementations:

:lang:jap — Implementation for APT/kapt based on javax.lang.model.**
:lang:ksp — Implementation for KSP based on com.google.devtools.ksp.**
:lang:rt — Implementation for reflection based on java.lang.Class and java.lang.reflect.*

2. :core:model — An abstraction of core entities in Yatagan: components, modules, and graph nodes. They’re built using language elements from lang.

3. :core:graph — This is where binding graphs are built using core models from :core:model. The graph can then be used for any operation, from checking for bugs to submitting for code generation or constructing reflexive implementation via java.lang.reflect.Proxy.

Here’s the general structure for Yatagan (it approximates how projects are divided into modules if you leave out the implementation details):

That’s how Yatagan guarantees that its backends will work identically:

Code for core models and the final binding graph is shared between backends.
Everything backend-specific is isolated in lang for implementation and the public backend-specific artifact.
Integration tests parameterized by the backend ensure that all backend-specific code works identically. Each test is launched for every backend to check behavior. Since the test code has nothing to do with implementation, it remains relevant even if you completely rewrite the implementation of a project component.

Implementing reflection

Supporting graphs during execution placed two limitations on the Yatagan API:

Components (@Component) and their factories (@Component.Builder) must be interfaces to be able to construct an implementation for them on the fly using java.lang.reflect.Proxy.
Components aren’t created with a direct call to a generated class like in Dagger (for MyComponent, you had to write DaggerMyComponent.builder().build()). Instead, a Yatagan object is used as a special entrypoint with methods for creating implementations of components: create() and builder(). It all comes together like this: Yatagan.builder(MyComponent.Builder.class).build() or Yatagan.create(MyComponent.class) if the component doesn’t have a factory.

The same approach is used for Dagger Reflect, resulting in the same interface requirements and the same special entrypoint. For Yatagan, that approach is standardized; the entryway class is always used since the generated component names are mangled. kapt/KSP implementation returns an error if you use an abstract class instead of the interface for a component.

Why Yatagan Can’t Support Hilt Even in Theory
There’s only one real reason: Hilt is an aggregating processor. While annotation processing engines offer that mode, which sacrifices build incrementality (as we saw in the Whetstone example), Java Reflection can’t aggregate. In other words, you can’t ask Java to return all of the annotated classes it finds in the runtime classpath. You can solve this problem with a bit of effort, but it will be extremely slow and lead to other issues. You can try to aggregate in compile-time and read the generated class lists in runtime, but that will abolish all the build speed profit gained with reflection usage in the first place. Ultimately, there’s no aggregate processing via reflection, so there could be none in any other backend otherwise compatibility would be broken.

Reflection and Android

Yatagan in reflection mode works for Android as well, though it’s fully compatible with it only starting with api version 24, where static methods in interfaces are natively supported without desugaring. So you have to make sure you’re using minSdk = 24 at least in your debug builds; otherwise static methods in @Module interfaces won’t be visible to Yatagan in reflection mode which may lead to “missing binding” errors.

What Happened to Support for Kotlin Properties in :lang?

Another interesting point is the support for Kotlin-specific linguistic constructions in language models. All three backends have a way of getting information about Kotlin language elements. For :lang:jap, that’s the kotlinx-metadata library, which can extract data from @kotlin.Metadata annotations. Then you don’t really need to do anything for KSP since the framework already models program elements in terms of Kotlin. For reflection, things seem pretty straightforward, too: you can use kotlin-reflect.jar, which can provide complete information about Kotlin entities via KClass.

It turned out that kotlin-reflect isn’t just as heavy as ~1.6 MB, which isn’t that bad for a debug-only mode, it also adds 3–5 seconds to app start times. Given that graph parsing at app startup in reflection already takes a few seconds as it is (for huge graphs like in our app), that’s just too much time. The slow initialization was caused by the library synchronously pulling information from kotlin_module files even when nobody asked it to. This information isn’t needed to just read properties inside classes. A bit frustrated, I decided to try the kotlinx-metadata library right in runtime rather than kotlin-reflect. Why not? @kotlin.Metadata annotations were available at runtime anyway. That worked and helped the situation, but not by much, since kotlinx-metadata uses a regular Java Service Provider Interface internally to locate and load metadata extensions. And that’s a very long process on a big classpath in an Android app. The problem probably could have been solved by correcting the code in the library and publishing it as kotlinx-metadata-jvm-only with a hardcoded extension for JVM, but it wasn’t worth the time and effort. Instead, I gave up modeling entities from Kotlin.

Note: companion object and object are still recognized, only with heuristics instead of actual Kotlin metadata.

Validating graphs at runtime with reflection

It is by no means superfluous to ask the question, how is error reporting done for graphs in reflection mode. As said, Yatagan is designed to provide uniform behavior across every backend. But here, there’s something different about the reflection backend.

Let’s talk about the behavior reflection has by default. If a graph is valid, there’s, of course, a well defined behavior. If a graph has an error, e.g. a missing binding, the backend behavior is undefined. In reality, the graph can throw an exception for critical errors with information about the error, but it’s possible that the component could execute without throwing any exceptions, resulting in what is ultimately classic undefined behavior. In other words, if the graph contains an error, the reflection backend’s default response is to consider it “ill-formed, no diagnostic required” (sorry for this infernal expression from the C++ standard’s world).

That decision was made to avoid sacrificing performance along the good code execution path — when graphs don’t contain errors. Reflection mode is designed solely for debugging, meaning it’s only appropriate for local developer builds. Complete validation should be run by code generators on CI servers.

But full validation can be enabled if needed! I introduced a special API that lets clients implement a special delegate for launching graph validation and printing out the errors it finds. When you create a root component, a validation task for the whole hierarchy is submitted to the delegate. The implementation can validate immediately, send validation to an asynchronous executor, or even postpone it. If an error is discovered while the component is working, the engine generates an await() call for the validation task to make sure it happens. An exception is thrown only after all the actual errors are reported.

In other words, you can get full validation in reflection mode, and it won’t undermine the time for constructing valid graphs. If there’s an error, we don’t really care how long the process takes since the app is going to crash anyway. All errors will still be printed to the console or logcat in the same format as for code generators, so in the end the backend uniformity is preserved even here.

Implementing KSP

Yatagan implementation on the KSP engine is experimental. Let’s take a look at why it can’t be completely stable as of now.

When I was still designing Yatagan around our project’s requirements, I decided that the backends, including KSP, should all behave the same way. That lets us switch between them without having to migrate or change the behavior.

Kotlin and Java are different languages, especially in their type systems. For frameworks like Dagger, type equivalence is incredibly important, and this is where Kotlin and Java differ in semantics. For example, java.lang.Integer isn’t equivalent to int even though they’re indistinguishable at the language level in Kotlin. Kotlin has List and MutableList, but they’re the same type in Java. Kotlin has information about nullability; Java doesn’t. Vanilla Dagger always judges in the context of the Java type system since it’s built on javax.lang.model.types.*, which models Java types. It only understands Kotlin to the extent kapt transforms it into stubs in Java.

KSP, while it supports Java, models language entities in terms of Kotlin. Java constructs are also modeled the way Kotlin sees them. But since we want compatibility with other backends and vanilla Dagger, and we want to generate code in Java, we needed to turn Kotlin types into their Java equivalents. That ended up being a challenge in KSP: information was almost completely missing in some places, while in others we had to jump through hoops, collecting information from different places at the same time just to ensure the end result behaved like Java. Even after all that, we can still find cases where KSP doesn’t behave like kapt or reflection.

At the end of the day, if you’re looking to work with code in terms of Java or at least generate Java code in the output, KSP is a bad option for frameworks. This is especially true for frameworks that extensively need to read program code and are sensitive to the type system. Why? Probably because KSP was designed firstly for Kotlin, and Java support didn’t get the attention it needed. Almost the entire JVM-specific API is annotated with @KspExperimental, which tells us that Java support as a whole is experimental and unstable. I understand what the Dagger developers went through to provide KSP support on their end, but they probably had it worse. They’ve had to support everything I’ve already mentioned and without the option of changing any behavior. Yatagan, on the other hand, is a new framework with no need for complete compatibility with Dagger, which made my life much easier back in the day.

That being said, KSP doesn’t work well outside Kotlin-only projects. If your project has a significant amount of code in Java, be careful with KSP. If your project is in pure Kotlin, give KSP a try.

Yatagan will eventually support Kotlin Multiplatform, which will require generating component code in Kotlin. We’ll probably roll out a kind of dedicated “pure-Kotlin” backend that won’t be compatible with the others and won’t convert all types to Java; that should make it easier to implement, although a separate set of tests will have to be written.

API Difference Between Yatagan and Dagger 2

The complete API compatibility table can be found in the documentation on GitHub.

The Yatagan API replicates the Dagger2 API in some places, but there are minor deviations in others.

It supports APIs from the following packages:

dagger.*
dagger.multibindings.*
dagger.assisted.*

The rest (dagger.android.*, dagger.producers.*, …) are not supported. I should also say a few words about dagger.spi: Yatagan supports plugins for validating graphs, only they should use the core API for Yatagan models rather than a dedicated one like with Dagger. You’ll find more information in the documentation along with the complete API correspondence table for Yatagan and Dagger.

Our Results

For our project, Yatagan accelerated incremental build time by 50–70% in different scenarios when using kapt (Yatagan kapt vs. Dagger kapt). Based on the measurements I took, KSP doesn’t have much to offer on top of kapt in out project performance wise. But it is worth noting, that KSP was broken most of the time for our project due to its bugs with Java support, so the results may be different for your project. Leveraging reflection gave us another 16–25% on top of kapt results since we could completely turn off annotation processing in the project’s bigger modules.

The code Yatagan generates also accelerated app startup in a number of scenarios by a few percentage points when we finished migrating. It’s hard to calculate the final number now since optimizations for Yatagan-generated code were written gradually. The benefit we saw was spread out, but we estimate it to be 5–10%. Given the situation, we call that a win.

Another Android project the company was working on, obviously without Whetstone, migrated from pure Dagger to Yatagan. For them, using RT instead of Dagger and kapt boosted their build speed by 40%.

What conclusions can we draw? The benefit you see depends directly on how much you use Dagger in your project and your Yatagan configuration.

Migrating from Dagger to Yatagan comes down to manually converting the API by replacing packages in imports. In some places, you’ll have to fix the annotation names, but by this point, your project could theoretically work. Practically speaking, you’ll still need additional fixes to make up for a few of the differences between Yatagan and Dagger. The more or less complete list of such differences can be found in the documentation.

Who Should Use Yatagan

If you have a small JVM/Android project using pure Dagger, and build speed is perfectly acceptable, and there’s no need for you to leverage conditional bindings in your DI, and though Yatagan could improve things even further, you may want to save the trouble of migrating to it.

Yatagan offers the biggest value in terms of local build speed if:

You have a lot of Gradle modules in your project with a single AP: Dagger.
Your modules with Dagger that contain root components (@Component) use other APs, in which case they should work in reflection mode or at least KSP, or can be otherwise removed from these modules.
Your developers agree to use Yatagan in reflection mode for debugging builds.

If that’s the case, then you can turn kapt completely off for quite a few modules, and this will greatly pay off in terms of build time.

If that third point is ignored and the team uses Yatagan in kapt or KSP mode, you’ll still get a tangible boost, it just won’t be as big. In that case, you can still opt out of kapt in modules without declarations of root components (@Component) since you don’t need Yatagan for them. KSP (vs. kapt) will give you more in specific cases (see the next section).

If the other APs from the second point support KSP mode, use it. Some frameworks support operating without code generation at all, and that’s a good option for debugging builds in reflection mode. If you use kapt-only APs with Dagger, try to move them to small, individual modules, if that’s possible.

If the first two points don’t hold true for your project , meaning most/all project modules have Dagger and at least one kapt-only AP included, you’ll get the least benefit from Yatagan. You won’t be able to pull the kapt out of any of your modules, and the biggest upside will be that you don’t have to compile generated factories. Builds will only be a few seconds faster. It’s still a benefit, but you may not consider migration for such numbers.

How You Should Use Yatagan

Let’s look at this in terms of Android projects. The reasoning is similar for pure JVM projects, too.

debugImplementation("com.yandex.yatagan:api-dynamic:1.0.0")
releaseImplementation("com.yandex.yatagan:api-compiled:1.0.0")
kaptRelease("com.yandex.yatagan:processor-jap:1.0.0")

You can go with reflection implementation for the app’s debug version if Dagger was the project’s last processor in kapt. In that case, we’ll see a huge improvement in builds without sacrificing start times too much.

Reflection is not an option for release builds though. For the time being, it doesn’t get along well with code shrinkers (ProGuard/R8), which would break it entirely.

If you don’t want to use reflection for some reason, use kapt or KSP. Here’s how to decide which to choose:

KSP only offers experimental support for Java, as practice shows, even if Google hasn’t said anything on the topic. There were frequent problems with it. This makes it risky for projects where a significant portion of the code is in Java.
You might try using KSP for pure-Kotlin projects. Bear in mind though that code is still generated in Java as of now. That necessitates an extra step for compiling Java and slows the build, though only slightly. So feel free to experiment.
If your module or project doesn’t have Kotlin at all, use APT. Everything will run like a dream, if you use pure Java.

In Conclusion

We will continue working on the product, and our plans include rolling out support for Kotlin Multiplatform with a dedicated KSP-only mode. But that’s only if we see interest in Yatagan elsewhere. If you have an interesting suggestion, feel free to create an issue with us on GitHub. We’re always open to feedback.