Enhancing 3rd-Party Library Support in GraalVM Native Image with Shared Metadata

Compiling applications that depend on 3rd-party libraries with GraalVM Native Image just got easier! There is now a GitHub repository that provides “reachability metadata” for popular libraries at

https://github.com/oracle/graalvm-reachability-metadata

When you use GraalVM Native Image to build a native executable it only includes the elements reachable from your application entry point, its dependent libraries, and the JDK classes discovered through static analysis. However, the reachability of some elements (such as classes, methods, or fields) may not be discoverable due to Java’s dynamic features including reflection, resource access, dynamic proxies, and serialization. If an element is not reachable, it is not included in the generated executable and this can lead to run time failures.

To include elements whose reachability is undiscoverable, the Native Image builder requires externally provided reachability metadata (previously known as configuration).

Up until now, users were required to manually define and maintain reachability metadata, not only for their application but also for the libraries on which their application depends. This can be a daunting task due to the widespread use of dynamic language features in 3rd-party libraries. However, most developers don’t know the internal workings of the libraries they use–which makes the definition of the correct reachability metadata a challenge.

To share the burden of maintaining metadata for 3rd-party dependencies, we have created the reachability metadata repository which enables users to share and reuse Native Image metadata for libraries and frameworks in the Java ecosystem. Ideally, library and framework maintainers will ship such metadata themselves, but they can also use the new repository to retrofit metadata for older versions of their software.

This new repository is integrated with GraalVM Native Build Tools beginning with version 0.9.12. To enable automatic use of the metadata repository for a Gradle project simply specify:

graalvmNative {
  metadataRepository {
    enabled = true
  }
}

The metadataRepository block causes the Native Build Tools to search the repository for the dependencies’ metadata, download the metadata, and automatically include it in the build. For further information about configuring the repository, see the build tools documentation for Maven and Gradle.

The metadata repository has been built with love as a collaboration between GraalVM, Micronaut, Spring Boot, and Quarkus teams. The initial set of libraries contributed by this collaboration can be found in the metadata folder.

Why a Common Repository?

Native Image is an emerging technology and library authors are sometimes reluctant to go the extra mile and maintain metadata for their libraries. Due to this, individuals and framework authors keep the metadata local to their projects which leads to: unnecessary repetition, incomplete metadata, and fragmentation of the community. Furthermore, some are using legacy libraries for which Native Image support would never be possible without externally provided metadata.

Here is also a community perspective on the topic from Brian Clozel, Spring Framework and Spring Boot committer: “With the reachability metadata, the GraalVM team kicks the native-image adoption into high gear. Spring projects have lots of integrations with the Java ecosystem, so it’s important to provide a wide support scope for the native experience. As a framework developer, maintaining all that metadata on my side is not sustainable nor efficient for the Java community. The reachability metadata repository gives an easy way for all Java developers to collaborate and bring the entire ecosystem up to speed”.

Metadata provided in the repository is not aimed solely at Native Image, although Native Image is the first use case. Reachability metadata can serve other use cases where knowing all of the used elements ahead of time is necessary (that is, a closed world is required).

Contributing Metadata to the Repository

Step 1: Collect Metadata

When the repository doesn’t have reachability metadata for a library, the easiest way to generate it is to execute the test suite of the library with the Native Image Agent enabled. The agent is installed together with native-image and can be used to identify elements used during application execution that cannot be discovered by static analysis.

Libraries using Gradle (and in the future Maven) can leverage the Native Build Tools to automatically enable the agent, to collect the metadata from the appropriate tasks/targets.

Gradle users can configure the agent by adding the following block:

graalvmNative {
  agent {
    defaultMode = “conditional”
    modes {
      conditional {
        userCodeFilterPath = “user-code-filter.json”
      }
    }
  metadataCopy {
     mergeWithExisting = true
     inputTaskNames.add(“test”)
     outputDirectories.add(
       “src/main/resources/META-INF/native-image/acme”)
     }
  }
}

To run the test task with the agent, run ./gradlew -Pagent test. This will generate metadata in the build output directory of the project (under build/native/agent/test).

The metadataCopy task copies, and optionally merges, metadata into the project specified by the output directories. This task can also be configured on the command line: ./gradlew metadataCopy –task test –dir src/main/resources/META-INF/native-image/acme

The conditional mode requires the users to pass an agent filter file that tells the agent which classes belong to the target library. This filter file has the same format as other agent filter files. Conditions will only be generated on classes whose names are matched by this filter.

A sample filter file (user-code-filter.json) for a library whose classes are all contained in `com.acme`:

{
  ”rules”: [
    {”excludeClasses”: ”**”},
    {”includeClasses”: ”com.acme.**”}
  ]
}

The collected metadata can further be filtered by setting extraFilterPath in the agent block:

graalvmNative {
  agent {
    modes {
      conditional {
        userCodeFilterPath = “user-code-filter.json”
        extraFilterPath = “extra-code-filter.json”
    }
  }
}

The filter excludes conditions and metadata that it matches. This allows easy filtering of irrelevant classes, e.g., classes used only during testing like Mockito mocks.

For other build systems, follow these instructions.

Step 2: Contribute Metadata

The best thing for the community is to include the generated metadata and build file changes into the library itself. Before creating a PR to the library:

1. Copy the metadata to the project sources under META-INF/native-image/artifact.id with metadataCopy.
2. Introduce a CI task that tests the library with the newly added metadata.

If the library maintainers do not accept the change, the next best thing is to open a ticket on the library’s issue tracker so the community can upvote this feature. In the meantime, you can open a PR to the reachability-metadata repository.

To avoid build breakage there are several rules for contributing to the repo:

1. The repo contains only metadata. This is the case as metadata can not break someone’s build–it just adds new elements to the image.
2. The metadata must be conditional. Unconditional metadata, although safe, can unnecessarily increase image size.
3. The metadata must be tested in the repo. The tests can be a subset of the whole test suite, but they need to exercise enough of the functionality to give confidence and avoid accidental breakage of the library’s core functionality.

For more information follow the contributing guide.

FAQ

Q: Can the metadata from the repo break my build?

It should not, the metadata only adds new elements to the image. If a breakage happens, please open an issue on the GraalVM issue tracker.

Q: Can the contents of the repo use Native Image internal APIs (e.g., substitutions)?

No, substitutions are intended only for the JDK functions.

Q: Does the metadata include any of the flags for the Native Image build, e.g., --initialize-at-build-time?

No, this could fragment the community as flags do not compose. This can break the build of some library combinations, while each separate library works on its own.

Q: Does the metadata include Native Image Features?

No, this could lead to a breakage of a build if misused.