IntelliJ: An Advanced Moveset for Shared Indexes
The beginning is a very delicate time
JetBrains IntelliJ IDE is a powerful development environment that was originally designed to write robust Java code but is now used for many languages.
However, as soon as you start working in IntelliJ, you will run into something that will halt all your progress in a moment.
It’s a one-of-a-kind phenomenon that’s spawned a slew of jokes. Yes, we’re talking about the dreaded indexing process.
Indexing? What is it for?
The short answer: to be able to support and aid you, dear programmer.
The IDE requires information about the project’s structure, dependencies, existing code base, and relations. It does this in two steps: first, it examines all available resources. Second, it constructs hashed indexes containing all the data about a particular resource and its relations.
Thanks to this, you can enjoy code completion, class linking and a cornucopia of other things that speed up your development process.
It all comes at the aforementioned price: when the indexing starts, better go enjoy your coffee break because there’s not much else you can do in the meantime.
The specific triggers that summon forth the indexing are:
- Project open
- Project structure change (plugins, modules, sometimes huge files)
- Changes in sources such as a repository branch change
“But I don’t want to waste my time and resources. Everything I wanted to do was halted. There must be a better way”, I hear you say.
And, lo and behold, here it is.
Shared indexes join the server
IntelliJ IDEA Ultimate comes with an inbuilt IntelliJ shared indexes plugin, which you may sample for free for 30 days with the Community edition.
Instead of manually creating indexes for a project, you can create them once and then download them to any IntelliJ instance that requires them. The crucial thing is that the download process begins during IntelliJ’s startup and is non-blocking. So what happens if the download isn’t finished before the local indexing begins? IntelliJ is smart enough to incorporate downloaded indexes into the indexing process in the middle, allowing it to finish faster.
The process of enabling and creating shared indexes for your project is well described on the IntelliJ help page: https://www.jetbrains.com/help/idea/shared-indexes.html. If you want to try it yourself, there is no better entry point.
Is it worth it?
Of course, it depends on your internet parameters. In my experience, a project that typically indexes in around 4–5 minutes is 80% faster when using shared indexes, at around a minute and a half. It’s a significant gain.
And if you want to try some shared indexes without a need to generate them, there is an option for you. JetBrains already pre-generates and hosts indexes for JDKs and Maven libraries. If you enable them, they will be automatically downloaded from JetBrains CDN and used during indexing.
Shared indexes: tricks & tips
Several key details about shared indexes are missing from the JetBrains documentation. Fortunately, I was able to piece together those missing parts via my own experience, and I am now able to share that information with you.
Shared indexes are fine for single project, but I work with a huge monorepo, there is no way I can index all of this projects
So, the thing is: you don’t have to index the whole project you are working on. Instead, you can index its subprojects. Or a project that it depends on. And you can provide more than one shared index resources in your .yml file:
sharedIndex:
project:
- url: <feed URL 1>
- url: <feed URL 2>
…
It is best to determine what you need to index by looking into the log/indexing-diagnostics directory in your IDEA directory. Here lies the HTML-formatted indexing report. Let’s see the statistics for the intellij-pants-plugin open source project — an IntelliJ plugin that enables pants support.
Here, you can see some interesting statistics. Amongst others, that modules and libraries are the main culprits of lengthy indexing. It also shows that using pre-generated indexes for IntelliJ sources and JDK 11 would save more than one minute of indexing — ⅓ of the total indexing time. You can speed up your process even without generating full indexes by identifying these key points.
I can’t generate indexes everyday for all the branches
What’s more, you don’t have to! If you supply an index for the exact code version, it will be used. IntelliJ uses a hashing function during the scanning step of indexing to see what may be rescued from previous indexes or downloaded shared indexes. And, in most cases, a lot of it can be reused. I worked with indexes that were six weeks older than the code version I was using at the time, and I couldn’t tell the difference.
What about 3rd party libraries outside of maven?
If you use Gradle to manage your dependencies, IntelliJ will still assist you in mapping Gradle libs to their Maven counterparts and downloading indexes. However, if you’re using a different build system, you’re out of luck.
Unless you’re an IntelliJ plugin developer because with IntelliJ 2021.3.3 and newer, you can do the matching yourself.
Intellij shared-indexes plugin exposes now an extension point indexing.shared.java.maven.mavenPackageCoordinatesInferenceExtension
that you can extend with your own implementation of MavenPackageCoordinatesInferenceExtension. A simple one-function interface matches the library path to its Maven metadata. We will use this one to enable support for pants.
