Taming feature flags

Hello, everyone! My name is Pavel Strelchenko, I’m an Android developer at hh.ru. In this article I’m going to talk about taming feature flags. If you prefer to watch a video instead of reading, go to our youtube channel. I will share more technical details in the article than in the video, so it must be riveting.

What are feature flags? These are regular Boolean flags which you use in an app to replace or enable/disable some functions. For example, one flag may change the button color, another — turn on or off the messenger inside the app.

If you have a small team and not so many feature flags, most likely you haven’t faced any problems yet. However, if the team is expanding, and the number of feature toggles is rising every day, you’re bound to face some difficulties. That’s what we’ll talk about.

Contents

• Feature flags’ problems
• Merge-conflicts solution
• Ways of feature flag collecting across all codebase
  * Manual collecting
  * DI frameworks’ features
  * Java Reflections API
  * Codegen
• Conclusions

The specifics of hh.ru

Before I tell you about the problems which we faced when working with feature flags, I need to immerse you in the specifics of our project.

First, we call feature flags “experiments”, because we’re constantly experimenting with our users while conducting a huge number of A/B tests. That’s how we’ve connected these two terms. So if we enable some experiment == we enable a feature flag. I’ll be using the term “experiment” throughout the whole article.

Secondly, a year ago we already had around 10 developers that were constantly writing a product code and almost every day creating new experiments in the codebase.

That’s where we came across some problems.

Feature flags’ problems

Our first implementation of experiment configuration can be characterized by the phrase “An abstraction never comes alone”.

An abstraction never comes alone!

Everything was complicated. We had an experiments-remote-config-core Gradle module where we described network requests for experiments’ lists and it’s caching. Also, there were two separate modules for our two main applications: experiments-remote-config-applicant and experiments-remote-config-employer, which contain all experiments for these apps. And a great number of feature modules depended on them.

We summed up the three key problems.

Merge conflicts in a general set of experiments

As I said before, we had modules where we described our experiments for different apps. It looked like a giant enum — one file where we described experiments line by line:

And, seriously, almost every day we faced merge conflicts: somebody merges a new feature into the ‘develop’ branch, and another developer creates the next one and merges ‘develop’ into a new feature branch. Both developers added new feature flags, so the conflict has to be solved in the same files.

Endless merge conflicts in one file

When merge conflicts happen again and again in the same files, you get a desire to solve them.

The whole app rebuild when adding an experiment

Apart from the public enum, which we changed when adding an experiment, we also changed the special public interface, which could be used in other modules.

When you change some public interface available to other modules, you change the ABI of your module. When changing ABI all depending modules shall be rebuilt. As we had a lot of feature modules that depended on the module with the list of experiments, we had to rebuild almost the whole app when adding another enum element.

It’s not supposed to be so.

Too much code for checking one flag

This problem is very specific for the hh.ru Android project — we’ve been writing too much code for checking one single experiment. Why did it happen so? For a long time, we considered the module with experiments to be a separate feature module, not core, which couldn’t be connected directly to another feature module, according to the rules of our codebase.

We had really complicated API ceremony for simple feature flag value fetching. It started in the feature module, where we created the interface for describing the dependencies of this feature module:

Then we went to the application module and defined Mediator-class, where the dependencies of the necessary feature module were implemented. There we fetched the experiments config from DI and took the flag value:

The experiment module had its own public interface (which I mentioned before) that we called from the application module.

Inside the experiment module, we implemented the necessary methods, which were mostly about checking the presence of the flag in the local cache:

Finally, enum with experiments was going through changes, as the new elements with the necessary keys were being added there:

The conclusions that we made:

• We need to have experiments in our codebase, we can’t just throw them away;
• We waste time on merge conflicts, that’s what we want to get rid of;
• We write too much code for a single experiment.

Let’s solve these issues!

P.S. To get more information about mediators usage, listen to “The Lord of the Modules” talk from Alexander Blinov, though I recommend watching an updated version of it on our blog.

Solving the problems

The first problem that we tried to solve was the merge conflicts. The technique I’m about to share with you is actually suitable for any situation, where you have merge conflicts due to adding lines inside a single file.

The technique is simple — you need to divide all the content added to the file into many files.

That’s why we introduced the Experiment interface and divided enum elements into separate classes implemented this interface:

According to our code style, we place each class in a separate file. It means that adding a new experiment won’t interfere with the experiments in other files, so the merge conflict problem is solved.

Next, we decided to simplify the process of checking the experiment: we created a special Kotlin object called Experiments, where we added a method for checking the presence of the experiment in the cache:

For more convenience, you can create an extension method on the Experiment interface, where the checking code will be even shorter:

To simplify over-complicated API ceremonies for fetching feature flags value, we decided to use our ‘experiments-remote-config-core’ as a real core module: we allow to connect it directly into feature modules.

The problem of long ceremonies in code was also solved.

Finally, solving the problem of rebuilding. Now each experiment is a separate class, so you can place them into SEPARATE modules and mark the experiment class with an “internal” modifier.

Thus, adding a new experiment won’t cause the rebuilding of half of the app.

However, a question arises: what should we do if the experiment needs to be checked in several different modules? The answer is simple: move the common experiment model into some core module, which can be connected to your feature modules. The main thing here is not to get carried away and not to put all the experiments in one basket. Otherwise, you’re doomed to suffer from multiple rebuilds.

Hooray, all the three problems are solved!

Collecting feature flags

We were faced with a new riveting task: how do we collect experiment classes scattered in our codebase into a single list?

Why do we need it anyway: our apps have a special debug panel for simplifying the app testing. This panel is available on debug build and minified debuggable build (we call it preRelease).

Debug panel and an open section for experiments

Inside the debug panel, there is a section dedicated to experiments — that’s where developers and QA engineers can change the value of the flags at any moment, to check some functions. Change the flag in the debug panel, reload the app, then check.

We got rid of enum, so we no longer have the function of getting the whole list of experiments embedded in the Kotlin language (we used to do it by ApplicantExperiment.values()). Plus, our server doesn’t send us the whole list of available experiment keys, it only sends the list which is active for a particular user. Therefore, you can’t just show the answer of the server in the debug panel.

What should be done?

That’s where the magic begins. There turned out to be a lot of remedies, and I’m going to share them with you. I divided these solutions into several groups:

• Make a list by hand
• Use DI-framework
• Think of Java Reflections API
• Generate the necessary code

In order to demonstrate this, I’ve prepared a Github repository. You can try each of the solutions in different branches.

Collect all items manually

This way doesn’t look reliable. We would face the same problem of merge conflicts, but this time they would be not in the enum, but in build.gradle files or when describing the list, where we would instantiate the models of experiments:

We came to where we started, but that won’t do.

On the other hand, this solution has some advantages:

• You don’t need any additional dependencies
• The implementation is simple

DI framework features

It turns out some DI frameworks can collect objects which are united by a particular feature (interface implementation, abstract class heirs, etc.) and make them into a list.

In particular, Dagger 2 and Hilt have a feature which is called Multibindings. Use it to collect objects either in Set or in Map with any keys.

How to?..

First, you create dagger modules in the necessary feature modules, where you describe the method marked by annotations @Provides and @IntoSet and the creation of the object:

After that you’re ready to inject the ready-made list in the necessary spot:

Here’s an interesting moment: when describing generic for Set we added @JvmSuppressWildcard annotation. Without it Dagger will be trying to find the wrong class, like the Java interface responding to “? extends Experiment” signature.

I said that DI frameworks were capable of collecting objects into lists. What about Toothpick/Koin? Unfortunately, nothing there.

The only thing these frameworks may offer is an issue in Github, where developers ask to add this possibility (issue for Koin, issue for Toothpick).

Thus, if you already use Dagger in your project — you’re lucky to be a user of the embedded feature for collecting the list of experiments. If you use a different framework — try other ways. HH uses Toothpick, so we continued the research.

Java Reflections API

Java Reflections API is an ability of Java language to recognize and change app behavior in runtime. Here I’m going to talk about several ways of collecting various parts code into a single list.

ClassLoader + DexFile

The first solution that I want to talk about is ClassLoader usage and Android Dexfile.

Before using some class in Java, you need to load it into JVM, that’s what ClassLoader does. DEX files are special files inside APK, which contain compiled code of your apps. The main point of it is that the bytecode format, used on Android, differs from the standard Java format, and this format is called DEX.

Having combined ClassLoader and DexFile, we get what we need — a list of separate experiments. Let’s break the implementation into several consequent steps.

The abstract scanner of the codebase

The first step is to create an abstract scanner of our codebase, which will have access to DexFile contents and filter the necessary classes:

We’ve got ClassLoader from the object of the current stream. Next, open DexFile, after having added the ‘package name’ of our app. Thus, we’ll obtain access to the list of all class names, available in the DEX file.

Now we have to filter the resulting list of names. Modern Android applications are quite complex systems that connect a ton of different libraries and work on the basis of an enormous codebase. Therefore, the list of class names can be very long. If you try to load each class inside DEX and analyze whether it is an implementation of the interface we need, it can be quite a long process.

That’s why first we filter classes according to their names with the help of the isAcceptableClassName method and only after that we can load the class and check whether it suits us — the check will be described in the isAcceptableClass method.

If the class suits us, we call the onScanResult accumulator method.

Particular scanner implementation

Let’s describe a particular scanner for our experiment classes by implementing the heir of ClassScanner:

In order to filter experiments by name, we set up an agreement on their names — each experiment class must have the suffix “Experiment”. In the isAcceptableClass method, we check that the class is the implementation of the Experiment interface and that the class being tested is not the interface itself — this is necessary so that we can create an instance of the class through clazz.newInstance.

The accumulator method simply adds instantiated experiments to the list.

Scanner usage

Launching the scanner, we get a list:

However, there’s a question: what will be displayed in the debug panel in the minified build type?

Having turned on Proguard, we find out that the experiment list is empty.

We turned on Proguard and now we are sad

Why did it happen so? Because the class names were minified, the “Experiment” suffix disappeared. There were no classes inside DEX corresponding to our condition. That’s why for this solution with ClassLoader and DEX to continue working, it was necessary to add one more line in proguard-rules:

It starts working after that.

Conclusions on ClassLoader and DexFile usage

Advantages:

• It’s a viable method, despite DexFile class being deprecated with API 26

But there are a bunch of drawbacks:

• This method requires agreement on naming experiment classes. My colleagues tended to consult me when they couldn’t find their experiment in debug panel because they forgot to put the necessary “Experiment” suffix;
• If you want to access the list of experiments in minified builds, you will have to enable keep names of experiment classes. The wrong part: potential competitors may spy on the contents of your APK file, see all the names of the experiment classes there, understand by the names what this experiment is about, and copy the logic to themselves;
• One year later, I still don’t understand why we chose this particular way of working with our experiments because there are other methods, easier and better =)

P.S. Speaking of competitors: in order to make the reverse of logic more difficult, one obfuscation of class names is not enough. It would be great if the keys of the experiments were some numbers that don’t reflect the whole idea of the experiment. The release APK should not contain any lines with literal descriptions of experiments.

ServiceLoader + META-INF/services

ServiceLoader is a special utility class in Java, that has been there since time immemorial. This class allows downloading the list of necessary objects (services) by using service providers.

The service provider is usually a text file located at /src/resources/META-INF/services/fqn.YourClass. Here,fqn.YourClass is the full name with the package of your common interface/abstract class. On each line of this file, you describe the provider — it looks like the full name of the class that implements the interface you need:

ru.hh.feature_intentions.onboarding.experiments.GhExperimentOne 
ru.hh.feature_search.experiments.AnotherExperiment  
...

By the way, it doesn’t matter at all what module you use for creating such a file, Gradle will add it to APK anyway.

Having described this file, you can use the ServiceLoader.load(YourClass::class.java) method, which will give you an Iterable with objects. The main rule for you is that your classes should have a default constructor without parameters:

Everything seems all right, but the problem is this configuration file with the description of the providers, because we stumble upon merge conflicts again, this time in the META-INF/services file.

Another disadvantage of this file is that it does not support auto refactoring in Android Studio. If you want to change the name of the experiment class or move it to another package, then you need to remember to correct its name in META-INF.

Conclusion:

• No additional outer dependencies and simple realization
• Drawbacks: the problem of manual filling of META-INF file.

Generating META-INF/services file

It seems rather logical to try and automatically generate META-INF/services file so that you don’t suffer from merge conflicts. There are several libraries for this, in particular:

• Metainf-services
• ClassIndex

The first is a bit outdated, it creates a lot of warnings in the app build-log, which are related to Java language versions.

You don’t get these warnings if you use the ClassIndex library. How should you work with it?

• Connect it via compileOnly + kapt;
• Attach the @IndexSubclasses annotation to the necessary base interface:

• Connect the library to each feature module, where there are models of experiments for their indexation:

After that, we use ServiceLoader.load method, get the list of experiments, and enjoy life.

However, there are some tricky parts.

ClassIndex adventure

In the video, I was saying that using the ClassIndex library for collecting classes in the codebase seemed to be the most appealing of all options. There’s almost no impact on the speed of build, no requirements for experiments class naming, and no specific rules for Proguard. So I decided to refactor our project from the ClassLoader approach to ClassIndex.

However, there were two interesting moments.

One of the disadvantages of ClassLoader was the need to save the names of experiment classes in a minified APK. What can this lead to? To the fact that an evil competitor may easily detect all the toggle classes inside your application, understand the logic of experiments by class names, copy it, and so on.

Of course, I would like to avoid a similar problem when using ClassIndex. But here’s the problem — based on the result of its work, ClassIndex generates a META-INF/services/fqn.of.your.class.ClassName file, that gets into the final APK. This file lists all the names of the experiment classes. After R8 processing, the already minified class names will be listed, which will make the job easier for potential attackers. What can be done?

I found only one reliable way to disable META-INF file generation: not to add ClassIndex dependencies into library modules where we define the experiments models. In this way, the annotation processor does not see that some class is the heir of the indexed interface — and does not add its description to the final META-INF service.

To do this, I added a simple Gradle parameter. Depending on it, we either add ClassIndex to the library module or not:

It is inconvenient to write this in each module, so you can take it to the convention plugin and connect the plugin to the library module.

It doesn’t affect local development in any way — product developers should not add any additional flags or change their usual flow of work in any way. But for building a release APK on CI, we can easily set a parameter:

./gradlew :headhunter-applicant:assembleHhruRelease -PdisableIndex

Thus, we have gotten rid of two problems with ClassLoader:

• We don’t have to get any special agreement on naming experiment classes;
• We don’t have to save the names of experiment classes in minified APK, ad it will be harder for an evil competitor to get all information at once

Profit.

The second interesting moment with ClassIndex was enabling incremental compilation for this library. Unfortunately, in May 2022 the latest ClassIndex version doesn’t support Gradle incremental compilation. We created PR into the Github project to enabling of this feature, but it is still not merged. So we created our own fork for it. The repository is public, but we created artifact only for internal usage.

Still, where are the adventures?

The adventures happened while I was looking for a viable way of disabling META-INF generation.

First I decided to try an option regarding AGP, which can remove any files from the resulting APK. You can do it using the PackagingOptions object, which you can get access to by the DSL method packagingOptions. We set the application modules via convention plugins, so I added a small code block there:

Thus, even if you write such an exclusion, the final APK will still contain this file. I don’t quite understand why it happens so, because the annotation processor is supposed to start working before packing files in the final APK.

• PackagingOptions object somehow didn’t resolve the situation. I thought it could be attributed to my mentioning the minified name of the experiment interface. I added proguard rule, which allowed me to save the name of the interface (Experiment), but nothing changed;
• I tried to delete temporary files META-INF, which appeared in different modules, while the annotation processor was working. I noticed that when I connected kapt and ClassIndex dependency in each module, there appeared a file for ServiceLoader in the fold build/tmp/kapt3/classes/release/META-INF/services. It contained the FQN of the experiment class. I tried deleting those temporary files before building the final APK (via TaskProviders available in android.applicationVariants), but it didn’t help;
• Hypothetically, by forking ClassIndex and adding a flag for annotation processor there, it would be possible to disable META-INF file creation for the final APK, but it’s a whole different story.

Conclusions on ServiceLoader and Meta-INF

Advantages:

• It’s the easiest way I ever found;
• No dependencies are needed in runtime.

A sort of disadvantage: you may need to increase the kapt workload, but just a bit, so it won’t affect the build speed seriously.

Reflections / Scannoation / Classgraph / etc

The third way inside the Java Reflections API direction is to use various libraries. When you try googling something like “java Collect classes with annotation across the codebase”, you will find a huge number of libraries related to the Reflections API.

I didn’t take into account absolutely everything, but looked at those for which there was good documentation and a lot of information on StackOverflow — these are Reflections and ClassGraph.

I’ll save you time and just say that these libraries out of the box don’t work for Android. Because Android has its own bytecode format when these libraries try to generate something, there are runtime crashes, it won’t work to use them in runtime.

On the other hand, in these libraries’ repositories on Github, you can find issues where developers are trying to find a way to adapt them for Android. You won’t find clear instructions there, but there’s a description of some methods that you could use — the list of the classes you need can be collected at the time of compiling your applications. When the compilation is complete, you can use the library in your build scripts, generate an intermediate file, and then read it in runtime, so everything is going to be amazing!

Actually, no. Well, at least I didn’t manage to do it in a reasonable amount of time, maybe you’ll have better luck.

Codegen / Bytecode patching

This is the last direction I wanted to talk about — the possibilities of code generation.

Two ways can help in solving our problem:

• Write your own annotation processor;
• Modify the bytecode

You can write your own annotation processor, but what’s the point, if you already have the same ClassIndex that will do exactly what you need. However, the second possibility looks interesting.

There is an ASM framework for bytecode modification. After your application has already been compiled, the resulting bytecode is received, and you can modify it as you need.

Colleagues from the Joom have already written Colonist library, which was ideally suited to our needs. Fun fact: the article about Colonist was published literally the next day after the start of my research.

Despite the fact that the library has not gotten over its alpha status, it can be used. How to do it:

• First, you will need to connect the library to your application. You can read more about this on Github:

• Secondly, we declare an annotation — ”colony”. A colony is a place where we will send “settlers”, that is, some kind of accumulator method that will collect the classes we need according to the rules that we set using tons of annotations:

We have described the ExperimentsColony colony annotation, into which we will let certain settlers — heirs of the Experiment class (that’s true, from the abstract Experiment class). We also inform Colonist that we will process settlers through a special callback.

• Then you have to describe the class for colony/collector:

We mark it with the colony annotation (@ExperimentsColony), and add a method for receiving settlers (marked with the @OnAcceptSettler annotation), done!

What happens under the hood: if you compile the APK to look at it using the jadx-gui utilities, you can see that the collector code is different from what you wrote. It’s the magic of changing the bytecode, and now there is a special piece of code that collects the heirs of the desired class and puts them into a specific method.

Checking the final bytecode via jadx-gui

• The last thing to do is use the colony

Conclusions on Colonist

Advantages:

• It’s a viable solution, although you need the alpha-version of the library

Disadvantages:

• You can’t really debug the method which interacts with settlers, because IDE doesn’t know anything about your manipulations with bytecode;
• The library contains true magic under the hood. If a usual annotation processor is pretty clear and you can touch the generated code, here it’s full magic beyond Hogwarts (if you want to figure out how ASM works — there’s a small documentation volume);

Let’s summarize

• First, you don’t have to suffer from merge conflicts, because it’s possible to avoid them;
• There are many ways of collecting models scattered in a codebase in a single list
• If we had Dagger, this article wouldn’t be here =)
• A small tip for you: don’t create extra abstractions, make your and your colleagues’ life easier.

I’ll be happy to receive your questions and additions in the comments below.