GeoTools On Java 9

CodeFX Weekly #31 — 18th of August 2017

Hi everyone,

A project I’m migrating to Java 9 (Cadenza, well summarized as “Google Maps meets Excel”) uses GeoTools, “an open source Java library that provides tools for geospatial data”. It seems to be a neat lib (haven’t really used it myself yet) and I spent some time with it because it found a very ingenious way to fail on Java 9. Let me tell you about that…

I send this newsletter out every Friday (well, this time it was Sunday). Yes, as an actual email. Subscribe!


Project of the week: GeoTools

Since we’re getting to know GeoTools anyway, I picked it as project of the week and decided to lead with that. Here are the project’s core features:

Definition of interfaces for key spatial concepts and data structures

  • Integrated Geometry support provided by Java Topology Suite (JTS)
  • Attribute and spatial filters using OGC Filter Encoding specification

A clean data access API supporting feature access, transaction support and locking between threads

  • Access GIS data in many file formats and spatial databases
  • Coordinate reference system and transformation support
  • Work with an extensive range of map projections
  • filter and analyze data in terms of spatial and non-spatial attributes

A stateless, low memory renderer, particularly useful in server-side environments.

  • compose and display maps with complex styling
  • vendor extensions for fine control of text labels and color blending

Powerful schema assisted parsing technology using XML Schema to bind to GML content. The parsing / encoding technology is provided with bindings for many OGC standards including GML, Filter, KML, SLD, and SE.

GeoTools Plugins: open plug-in system allowing you to teach the library additional formats

  • Plug-ins for the ImageIO-EXT project allowing GeoTools to read additional raster formats from GDAL

GeoTools Extensions_: Provide additional capabilities built using the spatial facilities of the core library. Extensions provide graph and networking support (for finding the shortest path), validation, a web map server client, bindings for XML parsing and encoding and color brewer!

GeoTools Unsupported: GeoTools also operates as part of a wider community with a staging area used to foster new talent and promote experimentation. Some highlights are swing support (used in our tutorials!), swt, local and web process support, additional symbology, additional data formats, generation of grids and a couple of implementations of ISO Geometry.

Now off to my problem!


GeoTools on Java 9

Registering factories

You see, GeoTools has a plug-in system that allows users to extend the library. Dutifully eating its own dog food, it amply uses that API itself, and as a consequence GeoTools is useless without it. Now guess which part stopped working on Java 9…

The plugin-systems has a FactoryRegistry at its core, which allows registering and requesting factories by their type. Furthermore, these factories are categorized by a common supertype. For example:

  • the category FruitFactory.class has a BananaFactory and a MangoFactory
  • the category StoneFacory.class has a DateFactoryand the same MangoFactory instance

To express preferences (I think) it is possible to declare a partial order across factories without having to provide a proper Comparator. So when iterating over all factories from a category, the iteration will respect the orderings that were defined between selected factories.

Instead of implementing all of that, GeoTools relied on the ServiceRegistry class from the package javax.imageio.spi and extended it with its FactoryRegistry. That worked fine in Java 8 - not so much with Java 9.

Brazenly breaking bonds

This happens, when you run GeoTools on Java 9:

java.lang.IllegalArgumentException:
org.opengis.referencing.datum.DatumFactory
is not an ImageIO SPI class
at javax.imageio.spi.ServiceRegistry.checkClassAllowed
at javax.imageio.spi.ServiceRegistry.
at org.geo...FactoryRegistry.
at org.geo...FactoryRegistry.
at org.geo...FactoryCreator.
at org.geo...ReferencingFactoryFinder.getServiceRegistry
at org.geo...ReferencingFactoryFinder.getFactories
at org.geo...ReferencingFactoryFinder.getCRSAuthorityFactories
at org.geo...DefaultAuthorityFactory.getSupportedCodes
at org.geo...CRS.getSupportedCodes

Weird, what does ServiceRegistry::checkClassAllowed do?

Err, ok? Why is this happening? Looking at the Java 9 documentation, you can find the comment:

The set of categories supported is limited to the following standard Image I/O service types:

  • ImageInputStreamSpi
  • ImageOutputStreamSpi
  • ImageReaderSpi
  • ImageTranscoderSpi
  • ImageWriterSpi

An attempt to load a provider that is not a subtype of one of the above types will result in IllegalArgumentException.

No explanation why that decision was made, though. Turns out the change was implemented in JDK-8068749, which says:

The lookupProviders methods in javax.imageio.spi.ServiceRegistry are wrappers around ServiceLoader.load. This is problematic for modules because the ServiceLoader.load method looks at their caller to check if the caller is in a module and if so, that the caller's module has the right "uses" in its module declaration. For now, then calling lookupProviders will cause ServiceLoader to check that ServiceRegistry's module (java.desktop) declares the "uses" and this will probably fail because the java.desktop module is unlikely to have it.

[…]

The alternative is to do nothing and just deprecate these methods. This option may not be too bad if we can establish that these methods are rarely, if ever, used outside of the JDK.

Long story short: It’s the module system’s fault. The result is that the ServiceRegistry API was broken rather brutally and without any warning. I wouldn't blame GeoTools for running into that problem, but it is a problem now and needs to be fixed.

On the beaten path

I was not the first to run into that problem. In fact it was November 2015 (wow!) when this was first discovered and an issue was created. Just two days later Vincent Privat opened a pull request, which proposed a solution. So why are we still having this problem now, more than 18 months later?

Vincent’s initial implementation simply copied the JDK source code. That’s a pragmatic solution and good enough to get things going locally, but GeoTools couldn’t merge the PR for legal reasons: Open JDK code is licensed under GPL v2 with the class path exception, GeoTools under LGPL.

A new PR was opened but it had similar problems and then a slow discussion with some messages going back and forth every other month slogged through 2016. From the outside it looks like Vincent’s solution worked too well for him to rewrite his solution from the scratch and the GeoTools maintainers didn’t feel the pain.

I did, though, because GeoTools is one of three major blockades in our Java 9 migration and I wanted to clear it. So: fork, clone, branch, remove extends ServiceRegistry, follow the compile errors and, et viola, a few hours later I had a GeoTools version that works perfectly on Java 9 and doesn't use any OpenJDK code. I opened a pull request, but I see first signs of the waiting game beginning again. Lets see how long it takes to polish (needs documentation) and merge it.

There’s not much to say regarding the implementation details. The partial ordering was interesting but 80% of the solution already existed, so I didn’t have to think too hard, just extend it a little but and clear a few bugs.

Oh yeah, one thing I did was deprecate all the methods that come from the ServiceRegistry API because they talk so much about providers (e.g. registerProvider) and create variants whose names sre more suited for a FactoryRegistry (e.g. registerFactors). I also started to replace the use of iterators with streams and the many legal null returns with Optional but that proved to be difficult in quite a few edge cases and I stopped to find out whether the maintainers are interested in that. Let's see...


Shots

  • Bohemian Haskell
  • looks like Bannon just left the White House — fuck yeah, one lunatic less

so long … Nicolai


PS: Don’t forget to subscribe or recommend! :)

A single golf clap? Or a long standing ovation?

By clapping more or less, you can signal to us which stories really stand out.