JSR 376 Ballot and Automatic Modules

CodeFX Weekly #18 — 12th of May 2017

Hi everybody,

this weekly is all about JPMS. The module system has seen a lot of publicity in the last week and I want to discuss the most obvious topic, the failed JSR 376 ballot, and also give some background to a recent technical development, a seemingly small change in automatic modules.

Before I come to the JSR 376 ballot, I have another vote to report on, though. A clear 80% majority of the responses to last week’s survey said that I should include the occasional news item, which I will do henceforth. I will of course continue to blab about other things, too, but not this week because the ballot and the discussion of automatic modules got so long.

I send this newsletter out every Friday. Yes, as an actual email. Subscribe!


The Vote That Didn’t Pass

I’m sure you’ve heard Tuesday’s big news: The public review ballot for JSR 376 did not pass! That leaves is with a lot of questions so let’s dive right in.

Who Voted What?

Let’s start off with a quick summary of the votes and some of the more interesting reasons that were given. The first vote came from IBM and established the mood of many to come: a “No” with a reference to somewhat nebulous technical reasons and the desire for more consensus in the expert group (EG).

The JSR 376 Expert Group and the public have raised a number of reasonable issues and concerns with the current public review draft of the specification that warrant further discussion and resolution. […] IBM would like to see closer consensus amongst the entire Expert Group before this specification proceeds to the next step.

Similar opinions, sometimes with more specific reasons, were expressed by Werner Keil, Hazelcast (problems for build tools), Software AG (automatic modules, cyclic dependencies, package isolation), Credit Suisse (automatic modules, reflection), SAP (insufficient specification), London Java Community, Ivar Grimstad, Twitter (package isolation), Eclipse Foundation (insufficient specification), Hewlett Packard, and Tomitribe.

Given their recently published concerns, Red Hat of course voted “No”. They gave a little more detail than most other members, though, for example by clarifying their focus:

from a middleware/SE developer perspective we believe that Jigsaw does not fulfil its original goals of being a module system which can be used by the likes of Java EE

Red Hat explained that they proposed solutions to the mailing list, which were often rejected, and also mentioned the lack of community consensus.

Many members who voted “Yes” gave no reason, a few stated that they shared some concerns but were basing their agreement on the assumption that they would be fixed before the next ballot: Oracle (surprise!), Intel, NXP Semiconductors, Goldman Sachs, Azul, MicroDoc, Gemalto M2M, V2COM, SouJava (clearer specification), and Fujitsu.

I find it important to note that many members, starting with SAP, lauded the work of the Jigsaw team and thus tried to improve the mood (or soften the blow).

We absolutely recognize the tremendous achievements and the great work that has been carried out until now — by the expert group members as well as (and especially) by the spec lead himself.

Many members also noted that the progress in the recent weeks gave them confidence that things will be resolved soon. I find that rather curious and will come back to it later.

About That Consensus…

The most often cited reason for a “No” was the lack of consensus within the EG. I don’t quite know what to make of that. Consensus is nice, of course, but is a lack thereof really a reason to vote against something as technical as a Java Specification Request? Why base your vote on whether people agree on something that you have all the resources to check for yourself? I have to admit, it sounds lazy.

What I can absolutely understand are technical concerns but again I find those votes lazy that just say “there are issues”. I mean, Jigsaw brings a module system, something that is by nature strict, into an ecosystem that for two decades didn’t give a fuck about any boundaries between artifacts (and suffered the consequences) — how can there not be issues?

So if a member thinks some design decisions were wrong, I would expect them to name them and articulate their stance on it — not necessarily as part of their vote but somewhere at some point.

Examples of this are Red Hat and IBM, who have long participated in the discussion on the mailing list and made clear which technical decisions they disagree with. That’s how I think an open process is supposed to work and even though I disagree with many of their opinions they expressed them consistently and in a way that allows the Jigsaw team to act on them if they think that’s prudent.

What Happens Now?

The JCP is very precise on this:

If the Public Draft Specification Ballot fails, the Expert Group will have 30 days to update the draft in response to the concerns raised by the EC and to submit a revised version to the PMO. If a revised draft is not received within 30 days, the original decision by the EC shall stand and the PMO will declare the JSR closed.

So Mark Reinhold, the specification lead, has 30 days to change enough of the module system to sway the critics before calling a reconsideration ballot, where JSR 367 will pass if it gets 2/3 of the votes that were cast.

Will The Next Vote Pass?

As I said, many “No” votes were lacking specificity, so it is hard to predict under which exact circumstances they might change. Most apparently refer to the wider criticism that Jigsaw solves too few actual problems while introducing new ones.

After reading the different accounts (particularly interesting are David M. Lloyd’s for Red Hat and Tim Ellison’s for IBM) I think it is fair to say that most votes would go “Yes” if the following were to happen (unless someone on the EG disagrees, of course, because “consensus”):

  • isolate modules (which would also allow multiple versions); likely by having a class loader per module
  • allow cyclic dependencies
  • improve support for integrating other module systems (often referred to as layer/module primitives)
  • improve migration story around automatic modules
  • give reflection its superpowers back

Regarding the validity of the Jigsaw team’s design decisions underlying those (and other) JPMS traits I refer you to a great post by Rafael Winterhalter, which puts many things in perspective and with which I full-heartedly agree. Here I want to discuss how likely I think it is that these things will change within the next 30 days:

  • module isolation via class loaders is utterly unrealistic because it is a lot of work to get right and causes too many compatibility concerns to be ironed out in one month
  • the deliberate decision to forbid cyclic dependencies is unlikely to be reverted (because it is a good decision)
  • the support for other module systems might happen but is currently being fought by Mark Reinhold because it would expose module system internals that were not designed for wider use
  • the issues around automatic modules seem to have been resolved
  • the fight between reflection and encapsulation is over and reflection has lost; no way is the Jigsaw team going to go back on that decision

So I don’t actually think much will change within the next 30 days. But hasn’t there been so much progress over the last weeks? Regarding that: no there hasn’t. There was a great decision regarding automatic modules (and I’m going to get into that right away) and lots of small clarifications but I see little evidence that speed has picked up recently — on the contrary, if it changed, it was more of a slowing down.

Going out on a limb I want to dare a partial prediction: Red Hat’s and IBM’s concerns will go largely unaddressed and the public discussion will focus on a few small things that people will fight over — in the end both will vote “No”. I think that’s fair because they are evidently concerned.

What will the rest of the committee do? Without appeasing Red Hat, there won’t be much more consensus than there is now and with the larger quarrels with Jigsaw unabated (particularly module isolation), the technical reasons for their “No” will still stand. So the real question is this: Are the members sufficiently convinced of their given arguments to kill JSR 376?


Update: Mark Reinholds Ask the Architect session at Devoxx UK partly confirms my assessment:

Automatic Module Names

So what was that big change that showed what great progress is possible? It has to do with automatic modules and how they get their names… To make sure everybody can follow along I’ll start with why automatic modules exist and what they are.

Automatic Modules As A Migration Mechanism

The JPMS has the goal to make the configuration of Java applications more reliable by giving the JVM a fighting chance to verify at launch time whether all dependencies are present. To that end, a module can express its dependencies with requires clauses, listing the other modules it depends on:

module java.sql {
requires java.base;
requires java.xml;
requires java.logging;
// only dependencies are shown
}

There are two properties of the module system that become important in the context of migration:

  • a module can only use code from modules that it depends on
  • a module can not depend on the chaos of the class path

So what happens when not all of a project’s dependencies were already modularized? Since a module can not depend on unmodularized artifacts on the class path, every project would have to wait until every dependency is a module — that would take forever!

The solution to that is that regular JARs can be put onto the module path (a concept paralleling the class path but meant for modular JARs), where they get turned into so called automatic modules. Automatic modules get a name, which means other modules can depend on them, and can read the class path, turning them into a bridge from modularized JARs to the class path.

Stick a pin into that — we will come back to it in a moment.

Stable Module Names

The module system is really strict about reliable configuration and it will not launch if a dependency is missing. This makes it problematic for a module to change its name from one release to the next because then all modules depending on it either…

  • … can’t work with the new version because it has a different name and would hence not fulfill the dependencies in the old version or
  • … update their own declaration and thus release a new version themselves, meaning they can no longer work with the old one

That’s already bad enough but imagine what happens when a transitive dependency X changes its name to Y and your project’s dependency A requires it as X but B was already updated to Y. Now the graph must contain X and Y but that’s not possible, so the application can not be launched with these modules. Module name hell!

This makes stable module names a very high priority. Have a second pin? Stick it in here.

Unstable Automatic Module Names

I handwaved you past a detail of automatic modules: How their name is chosen. By definition they have no module descriptor, so where does the module system take it from? The answer used to be: from the file name. So com.google.guava-21.0.jar becomes com.google.guava and guava-21.0.jar becomes guava.

Coming back to what we just discussed, what do you think how reliable of a module name the file name is? Not a terribly reliable one, right?! On the contrary, because all ~/.m2-based build tools have file names like guava-21.0.jar but module names are supposed to be reverse DNS (like packages) they are almost guaranteed to change once a project gets modularized.

This essentially brings us back to square one, where a module an only be modularized once all dependencies are modules. (This is not entirely true because as long as the dependency on the automatic module is not published, it can easily be edited but that obviously only helps application, not libraries or frameworks). Once again, the ecosystem would have to modularize from the bottom up.

Manifest to the rescue!

The big change that was bandied around so much was that Mark Reinhold accepted a suggestion (or rather a small variation of one) that was proposed a couple of weeks ago (and originally declined): If a JAR’s manifest defines the attribute Automatic-Module-Name, then that becomes its name instead of the file name.

This seemingly small change allows projects to claim their module name early (actually, already now), which will have a big effect. Now, instead of having to wait for true modularization of all of your dependencies before turning your project into a module, you can do so as soon as all of them declare the Automatic-Module-Name artifact.

This is still no perfect solution. One drawback, for example, is that the attribute can of course not be added to artifacts that were already released. If you deliberately support old versions of your dependencies to not force your clients to update, then you have to weigh sticking with that commendable strategy against modularizing your project.

The other thing is that even though this mechanism reduces the risk of module name hell, it does not drop it to zero — there is still the chance that somebody releases a module that uses a name derived from the file to refer to an automatic module.

Instead of fighting the inevitable I proposed a mechanism that could allow to handle the situation when it eventually occurred. With an aliasing mechanism that would allow a module to stand in for one with another name this as well as another few problems would be solved. But it would of course have been a much deeper change than just adding an attribute, which is why it was declined.

Was it the pressure?

From reading comments from the EC, on Twitter, and on Reddit I get the impression that most people think that Mark came around on this because of how bad the vote was turning out. That is surely possible but for the record I want to point out two things:

  • Jigsaw has been a moving target for years now and there was no sign of slowing down during the ballot
  • lots of criticism or suggestion by the community has initially been summarily declined before being suddenly accepted weeks or even months later (most prominently, keeping Unsafe accessible)

Of course the timing of Mark Reinhold’s relenting is at least suspicious but not damningly so and on a timeline of what-happened-when-in-Jigsaw this decision would not stand out at all, so I give the Jigsaw-team the benefit of the doubt and assume the decision was made for technical reasons.

And that’s it for this week! No project of the week, no shots, I’ll keep all of that for next week.

so long … Nicolai


PS: Don’t forget to subscribe or recommend! :)