Making Clojure and Java libraries with static resources work

I’m still pretty new to JVM land. After some experience with JRuby and now running Clojure in production for almost two years I think I know my way around some of the Java interop stuff, tuning JVM and learning that JVM’s regex engine doesn’t work like in other languages (performance wise).

However there’s another real world concept that every real software project will run into: dependencies. Clojure apps which depend on libraries written in Clojure are easy — Leiningen solves all the problems. Pulling existing libraries written in Java from, let’s say Maven Central is also ok — although more complicated things like Spark are problematic sometimes.

Where it got really hard was how to use a library written primarly in Java, with a Clojure layer on top (to hide the Java stuff ;-)) and make it usable in a Clojure project.

Expectations

For example — when packaging a Ruby project into a gem, it’s pretty safe to assume that if your library ships with a resource file (for example a html template), it can be read like this:

class Foobar
TEMPLATE = File.read(File.expand_path('./tmpl.txt', __FILE__))
end

This pretty much guarantees that if foobar.rb and tmpl.txt are in the same dir, no matter where and how the gem is installed or if we're just running the code via ruby ./lib/foo/foobar.rb.

In Java/JVM land things are not so simple.

Above snippet would work, however if we’re creating and distributing a library as a jar file things are getting complicated.

Real world

My scenario was this:

  • I have a service written 100% in Clojure (let’s call it foo, obviously)
  • I have a library bar, also 100% in Clojure
  • Library baz is 99% Java and 1% is a Clojure wrapper, additionally baz needs static resource files to work (configuration & data file for a NLP model)

foo depends on bar and baz.

As I’m using Bintray to host a private Maven repo, things are pretty simple. With Leiningen all I have to do is add:

;; used for publishing as a lib
:deploy-repositories [["releases"
{:url "https://api.bintray.com/maven/repo/maven/bar/;publish=1"
:sign-releases false
:username :env/bintray_username
:password :env/bintray_api_key}]
["snapshots"
{:url "https://api.bintray.com/maven/repo/maven/bar/;publish=1"
:sign-releases false
:username :env/bintray_username
:password :env/bintray_api_key}]]

to project.clj and run lein deploy. This will compile everything, create a maven package and upload it to Bintray.

Then in foo's project.clj:

:repositories [["bintray"
{:url "https://repo.bintray.com/maven"
:snapshots true
:username :env/bintray_username
:password :env/bintray_api_key}]]
:dependencies [["bar" "0.0.1"]]

will make private libs available as dependencies.

So far so good

As one would expect baz the Java/Clojure lib proved to be a bit problematic:

  • extra resource file was read at runtime, and the code assumed it’s available under resources/db.txt
  • when deployed as a jar (even locally, using lein install) the file would get included
  • however using baz as a dependency in foo wouldn't work as the file's path would no longer be a file system path, but instead it would get turned into a resource.

My first approach was to convert all the code from simply reading files from paths to using resources:

// before
class SomeStuff {
private final db;
  public void SomeStuff(String pathToDB) {
db = new BufferedReader(new FileReader(pathToDB));
}
}
// after
// in tests
InputStream in = this.getClass().getResourceAsStream(pathToDB);
class SomeStuff {
private final db;
  public SomeStuff(InputStream db)
db = new BufferedReader(new InputStreamReader(db));
}
}

then in Clojure:

;; before
(SomeStuff. "resources/db.txt")
;; after
(require '[clojure.java.io :as io]
(SomeStuff. (-> "db.txt"
io/resource
io/file
io/input-stream))

I’ve run the test and pushed to our CI server. Everything works. Tested the code in REPL, all fine.

Neat

After lein install I've happily used baz code in foo and run the tests and...

billion lines of stacktraces
Caused by: java.lang.IllegalArgumentException: Not a file: jar:file:/home/vagrant/.m2/repository/baz/baz/1.0.7-SNAPSHOT/baz-1.0.7-SNAPSHOT.jar!/db.txt

Since jar files are just zips, I've peeked inside and db.txt was there. Both Clojure and JUnit tests were passing fine in baz so... what happened?

I’ve started checking out how other people do this since Googling didn’t help much. Very quickly I realized my mistake. You see java.io.InputStream knows how to deal with many things - not only Files but also... Resources.

So:

;; before
(require '[clojure.java.io :as io]
(SomeStuff. (-> "db.txt"
io/resource
io/file
io/input-stream))
;; after

(SomeStuff. (-> "db.txt"
io/resource
io/input-stream))

Seemed like a bit of a random change but:

  • tests in baz worked just fine
  • after installing to a local Maven repository foo pulled baz just fine
  • tests in foo worked as expected

Summary

TIL how to:

  • ship a mixed Clojure/Java project as a lib
  • that lib has some resources (that are not code)
  • and how to use all that in another Clojure project