Debian packaging of a pre-compiled java application

Sad story: we have a java application, which is free and available for download as ‘tar.gz’, but we don’t want to use ‘curl-install’ on our production server from somewhere-in-Internets. There are two ways to deal with it: to move the archive to a private http mirror or to package into deb.

Packaging into deb is much cleaner approach as I it provides clean mark for configuration files (conffiles), it puts application to /usr/share/app_name, it moves configuration file to /etc, and automatically install jre when neede. And, of course, it adds upgrade/purge/remove support.

My packaging isn’t a proper work, as a ‘build’ procedure has nothing to do with a real source code, but it is better then just ‘unpack and run’. Here I’ll describe how to do this in a reasonable way, without deviating from delivery pipeline at all.

Binary is a source

The trick is to declare that original application’s ‘tar.gz’ is the source code. This is a big fat lie, but dpkg could not distinct and no one care about true source code anyway in our case. In all those examples I’ll use “foo-cli” as a name for my application.

I started from creating my own git repo.

cd ~git/Packages
mkdir foo-cli
cd foo-cli
git init

Next I imported my ‘upstream source code’ (which is actually a precompiled binary with jars, shell scripts, example configs, etc):

gbp import-orig --upstream-branch=upstream /tmp/foo-cli-7.1.1.tar.gz

It created the branch upstream, unpacked tar.gz content into it, placed tag upstream/7.1.1 to it, merged it to the ‘master’ branch. Now we are ready to debianize it. Let’s create debian folder and start placing ‘important stuff’ into it.

gbp.conf

First thing we need to do is to configure gbp itself. All following changes are happening in the ‘master’ branch of the git repo.

debian/gbp.conf:

[DEFAULT]
upstream-branch = upstream
debian-branch = master
upstream-tag = upstream/%(version)s
compression = gz
pristine-tar = False
[buildpackage]
export-dir = ../build-area/

gbp will read those files before building our package. Important bits:

upstream-branch = upstream
debian-branch = master
upstream-tag = upstream/%(version)s

We can check with git tags command, that indeed, tag upstream/7.1.1 was created.

Debian format specifications

We need to say to dpkg which version of specification we are using and what kind of package we are building. There are two types of debian packages — native (debian-specific) and quilt (using upstream source code). We always build quilt packages.

debian/source/format:

3.0 (quilt)

(Don’t forget to create ‘debian/source’ folder).

Next, we need to specify which version of debian specification we are using (‘compat level’). We will use xenial level, which is around 9.

debian/compat:

9

(yep, just one digit “9” in the file)

Package declaration

There are three key files in debian packages for each package: control, changelog and rules. The last one will discuss a bit later, and for now let’s focus on control and changelog. Control defines everything about package: its name, dependencies, description, homepage, etc. The single thing which is not in control is version. It is calculated from the the changelog file.

debian/control:

Source: foo-cli
Priority: optional
Section: java
Maintainer: George Shuklin <g..@gmail.com>
Build-Depends: debhelper (>= 9), devscripts
Standards-Version: 3.9.8
Package: foo-cli
Architecture: all
Priority: optional
Depends: ${misc:Depends}, default-jre-headless
Section: java
Homepage: http://foo.example/
Description: Foo cli utility
 See homepage for details.
 This package repackage original compiled
 java binaries for foo and provides
 additional symlinks to /usr/bin

Key things here:

1. Name of the source package is the same as binary package. It’s ok.
2. Section java — because this is java application
3. Build dependencies are needed for jenkins-debian-glue to create a suitable environment for build. Normally they are checked before build by dpkg-buildpackage. We do not touch java here, so all we need is usual debian packaging tools.
4. We declare foo-cli binary package. It’s java, so ‘Architecture: all’. Please note, ‘any’ and ‘all’ are different things. ‘Any’ means we can build binary packages under any processor architecture, ‘all’ means our packet will work on all architectures without rebuilding.

Now, let’s create a changelog.

debian/changelog:

foo-cli (7.1.1-1) xenial; urgency=medium
  
  * Initial release
-- amarao <g...@gmail.com>  Thu, 03 May 2018 11:55:35 +0300

Key things here:

1. Version become 7.1.1–1. This is Debian requirement — we took 7.1.1 upstream package and made version 1 of our debianization. If we ever need to change our debianization, upstream version my stay the same (7.1.1), but our version would become 7.1.1–2.
2. ‘foo-cli’ name should match source package name.
3. xenial is our target distro.

Normally I create this file with dch utility.

License

cp LICENSE debian/copyright

Build rules

Now, finally, we need to describe how to build our package. This is done with makefile named debian/rules.

debian/rules:

#!/usr/bin/make -f
%:
    dh $@
override_dh_strip_nondeterminism:
   true

(use tab symbol, not four spaces, as make utility requires to use tabs)

WUTS SIS?

To be honest, I hate this part of debian packaging. There are so many complicated things happens here and nothing is obvious.

Whole packaging in debian happens by chan-calls to different targets in debian/rules in a specific order. I do not remember whole list, it’s described in dh manpage: build-arch, build-indep, build, clean, install-indep, install-arch, install, binary-arch, binary-indep, and binary.

Our makefile (debian/rules) says ‘for every target run dh that_target’.

dh is a wrapper over many other commands called helpers. Examples are dh_install, dh_clean, etc. Each such helper may call few more helpers for a specific job (like dh_manpages, dh_link, etc).

Before calling them dh (and all helpers) check if there is a corresponding override_HELPER_NAME. In our case we override dh_strip_nondeterminism (it called somewhere inside binary-indep stage).

Our rules file let everything to be don ‘by default helpers of dh’ except for dh_strip_nondeterminism. We override it with override_dh_strip_nondeterminism target which says ‘do nothing’.

Because we have many helpers called during that ‘dh auto cycle’, we can change their behavior with additional files. This will be described in the next section. For now — this small ‘rules’ file defines all build procedures. We can be sure in that behavior because we set compat to the specific version. Helpers behavior may change between versions, but for a given compat level we guaranteed to have the precise behavior forever (until this compat level no longer supported).

Minor tweaks (the main part)

foo-cli archive assumes we have everything in a single directory. Configs, modifiable data files, etc. It’s windows way 100%. We will change this.

First, we’ll specify our directories (we need to have in the file system to proceed).

debian/foo-cli.dirs:

/usr/share/foo-cli
/etc/foo-cli

(please not the file name foo-cli.dirs — it should match the binary package name).

Next, we specify where to install immutable files:

debian/foo-cli.install:

/drivers /usr/share/foo-cli/
/jars /usr/share/foo-cli/
/lib /usr/share/foo-cli/
/foo-cli /usr/share/foo-cli
/conf/* /etc/foo-cli

Please note, those pathes look absolute, but they are relative to the source package root.

We say that directory drivers, jars, libs need to be installed in /usr/share/foo-cli.
Then we say ‘binary file foo-cli should be in /usr/share/foo-cli/'
Then we say ‘content of the conf directory should be in /etc/foo-cli/’

And now a bit of magic. We have dh_links helper, and we’ll instruct it to make few softlinks:

debian/foo-cli.links:

/etc/foo-cli /usr/share/foo-cli/conf
/usr/share/foo-cli/foo-cli /usr/bin/foo-cli

It uses ‘ln’ syntax (“from to”). We are creating a link from /etc/foo-cli to /usr/share/foo-cli/conf, where our foo-cli expects to find its config. Configs are mutable, so we moved them away from ‘/usr’. The symlink allows us to have ‘mutable configs’ in the ‘immutable /usr’ where foo-cli expects to find them.

Next, we create a symlink to /usr/bin/foo-cli — our foo-cli binary (it’s a shell file which runs some jar with java). We have original binary in the original location — it uses relative pathes to find configs and libraries, but the symlink allows us to add foo-cli to the usual location for binaries (which is in the $PATH).

There is a bit of automation happens between scene (f.e. conffile is created automatically for all ‘/etc/*’ entries), but we are done.

Let’s commit those changes into git, and now we can try to build our package.

Building

$ gbp buildpackage --no-sign

gbp buildpackage --no-sign
...
gbp:info: Performing the build
 dpkg-buildpackage -rfakeroot -us -uc -ui -i -I
dpkg-buildpackage: info: source package foo-cli
dpkg-buildpackage: info: source version 7.1.1-1
dpkg-buildpackage: info: source distribution xenial
dpkg-buildpackage: info: source changed by amarao <g...@gmail.com>
 dpkg-source -i -I --before-build foo-cli-7.1.1
dpkg-buildpackage: info: host architecture amd64
 fakeroot debian/rules clean
dh clean
   dh_clean
 dpkg-source -i -I -b foo-cli-7.1.1
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building foo-cli using existing ./foo-cli_7.1.1.orig.tar.gz
dpkg-source: info: building foo-cli in foo-cli_7.1.1-1.debian.tar.xz
dpkg-source: info: building foo-cli in foo-cli_7.1.1-1.dsc
 debian/rules build
dh build
   dh_update_autotools_config
 fakeroot debian/rules binary
dh binary
   dh_testroot
   dh_prep
   dh_installdirs
   dh_install
   dh_installdocs
   dh_installchangelogs
   dh_perl
   dh_link
   debian/rules override_dh_strip_nondeterminism
make[1]: Entering directory '..build-area/foo-cli-7.1.1'
true
make[1]: Leaving directory '..build-area/foo-cli-7.1.1'
   dh_compress
   dh_fixperms
   dh_missing
   dh_installdeb
   dh_gencontrol
   dh_md5sums
   dh_builddeb
dpkg-deb: building package 'foo-cli' in '../foo-cli_7.1.1-1_all.deb'.
 dpkg-genbuildinfo
 dpkg-genchanges  >../foo-cli_7.1.1-1_amd64.changes
dpkg-genchanges: info: including full source code in upload
 dpkg-source -i -I --after-build foo-cli_7.1.1
dpkg-buildpackage: info: full upload (original source is included)
Now running lintian foo-cli_7.1.1-1_amd64.changes ...
W: foo-cli new-package-should-close-itp-bug
W: foo-cli: binary-without-manpage usr/bin/foo-cli
Finished running lintian.

We got two warnings. One is inevitable for non-debian packaging, second is my guilty negligence. I have no time to do man-pages. There are no other warnings or errors, that means we’ve done build this package right. (In reality I done few iterations changing small things until most of the warnings are gone).

Now we can set Jenkins on a new job to build and upload our package to the private apt repository using our usual workflow of jenkins-debian-glue and aptly.

Conclusion

We’ve presented a way to build deb-packages from binary pseudo-sources to allow seamless integration with existing delivery pipelines.