How Compatibility Breeds Complexity

CodeFX Weekly #28 — 28th of July 2017

Hi everyone,

today’s weekly is laser-focused on a single topic: How compatibility breeds complexity. It’s not a particularly novel thought, but it really struck me last week when I formalized how the Java module system handles illegal access to internal APIs: This got way more complex due to compatibility concerns.

You can find the most detailed account on how illegal access is handled in my book about the Java 9 module system but I also published a post that explains most of what you need to know: the Java 9 migration guide.

I send this newsletter out every Friday. Yes, as an actual email. Subscribe!

How Compatibility Breeds Complexity

The thought’s essence is straight forward: Solutions unfettered by past decisions can be simple and beautiful but once it has to properly cooperate with existing implementations, life gets much more complicated.

A Simple World

In the Java Platform Module System (JPMS), strong encapsulation is a pretty simple concept:

  • for static access (e.g. imports or fully qualified class names) only public classes and members in exported packages are accessible
  • for reflective access only public classes and members in exported packages and classes and members (of any visibility) in open packages are accessible

Everything else is inaccessible from outside the module. If an application absolutely has to use internal APIs, it can use the command line options --add-exports or --add-opens to export or open packages as needed.

So far, so good.

The Real World

Now we enter the real world, where Java is trying to be backwards compatible. The described behavior, mixed with the common use of JDK-internal APIs would mean that many applications wouldn’t run on Java 9 without extensive command line options. So after long discussions and a lot of lobbying, compromises were made.

(In the mail, what follows used to be a complicated looking nested list but Medium doesn’t support those, so I had to split it up, which is stupid, but here you go…)

On Java 9, illegal access is handled as follows:

  • the compiler prevents illegal access
  • the run time is more lenient

The run time differentiates between module path and class path:

  • code on the module path can not access internal APIs
  • code on the class path has more rights

Code from the class path is treated differently based on the kind of access:

  • static access is allowed
  • reflective access is allowed and the first such access to each packages results in a warning on the command line

Well, when I said “allowed” I forgot to distinguish by which Java version introduced the package that is being accessed:

  • access to internal packages that existed before Java 9 is allowed
  • access to internal packages that are new in Java 9 is prevented

You can configure the behavior warnings:

  • opening a package on the command line removes the reflective access warning
  • the option --illegal-access can be used to increase warning frequency (once per access) and detail (include stack trace) or outright deny illegal access

Wow, look at that explosion of concepts and cases! Before, you only needed to know about exporting packages and opening packages and that you can do that in the module descriptor and on the command line — a small, two by two package of concepts.

Now you also need to differentiate between compile time and run time, whether the accessing code comes from the class path or module path, static and reflective access, package age, and how --illegal-access and --add-opens change warnings. You're lucky if that conceptual package fits into a five-dimensional U-Haul.

I’m Not Just Complaining!

I’m not telling you all of this to complain about the decision that made this the default behavior. Yes, I disagree with it but that’s beside the point — I’m telling you this because I think it’s a great example for how compatibility breeds complexity.

When developing a new solution for a known problem from a clean slate, chances are you can come up with a simpler and more robust solution than the one currently in place. The module system strived for that and I think it does a good job at it.

But if you value backwards compatibility as high as Java does, then at some point you have to bridge from the old solution to the new and provide users with a path that they can walk to get there. And this is where the incongruence between two approaches, even if each of them is fairly straight-forward, can disfigure even the most beautiful solution.

What was made possible by decisions past still has to work even though it might go against everything your new approach stands for. In the module system, reliable configuration is king but on the class path reigned chaos that besides all its downsides would occasionally be used for good. Strong encapsulation is queen, but before Java 9 the ecosystem cannibalized the JDK for every last line of code.

How do you bring diverting and sometimes even diametrically opposed requirements together without ending up with a complex, disfigured beast that makes everything much worse than it was?

Some Thoughts on the Topic

In case you’ve been waiting for an answer to that question: I don’t have it. Here are a few thoughts:

  • When looking for a new solution to the known problem, wouldn’t it be better to ignore compatibility concerns? I prefer to create something nice and simple without getting bogged down in the sludge of the past.
  • Eventually the time comes to evaluate a proposal, though, and now it is very important to analyze how it handles compatibility concerns. Gracefully or does it turn into a quagmire? It pays off to be very aware of the tendency to stick to ones own ideas. It can be hard to let them go…
  • Maybe a different approach is (almost) as good as the proposal at hand but provides a simpler upgrade path?
  • Do the hard part first? Before implementing the beautiful new API, implement the compatibility layer/features. It’s way less fun but this way you reduce the risk of throwing away your work.
  • When integrating compatibility concerns, make sure you don’t stray too far from the original concept. Constantly evaluate whether you’re still true to the original idea. If not, take a step back. Maybe a different idea is better after all.
  • When bolting on compatibility features, make sure people don’t get stuck half way from old to new version, making everything worse than before. It is important that there is always a path and an incentive to take the next step towards the new solutions.

How Fares The Module System?

Of course I know next to nothing of how the Jigsaw team’s internal processes work but I can evaluate the outcome.

I really appreciate that the module system’s basic ideas, reliable configuration and strong encapsulation, were not watered down. With command line flags, configurations can be hacked and internal APIs exposed but I think that strikes a good balance between ease of use (not too hard) and deterrence (not directly available for libraries/frameworks; not comfortable to use for application developers).

The whole “allowing illegal access by default” idea does also fulfill my criteria quite well. It provides seamless compatibility (at least regarding internal APIs) at the moment but will push teams into the right direction by making the default value stricter over time, removing the more laissez faire options in the process.

Assuming not too many new warts blossom, we will eventually arrive at quite a pure version of the JPMS’ vision. And isn’t that something?


Apparently I was too disciplined to waste much time, so nothing to see here.

so long … Nicolai

PS: Don’t forget to subscribe or recommend! :)

Photo by Kletis Roy on Unsplash