One more thing: Haiku application bundles?

17 min readJul 16, 2019

TL;DR: Could Haiku gain from having proper support for Application bundles such as application directories (think Mac .app) and/or application images (think Linux AppImage)? I think they would be a worthwhile addition, and much easier to implement properly on Haiku than on other systems since most of the infrastructure is already in place.

A week ago, I discovered Haiku, a “shockingly good” desktop operating system. Since I have been interested in and a big fan of application directories and application images (inspired by the simplicity of the Macintosh) for a very long time, it is only natural for me to wonder…

Full disclosure: I am the inventor and original author of AppImage, a format for distributing applications on Linux that aims for Mac-like simplicity and puts application authors and end users in control (if you want to learn more, see https://en.wikipedia.org/wiki/AppImage and https://docs.appimage.org/).

What if we wanted to make AppImage for Haiku?

Let’s for a moment, theoretically, think about what would need to be done to have AppImage or something like it on Haiku. I am not saying that we should necessarily do it at this point, because the system that Haiku has in place is working amazingly well, but I find it an interesting thought experiment. Turns out also demonstrates the elegance of the Haiku system when compared to Linux desktop, where this kind of thing is terribly difficult (i.e., has already cost me over 10 years to get right — and counting).

On Macintosh System 1, each application was a single file that could easily be “managed” in the Finder. With AppImage, I have been trying to re-create this user experience for the Linux desktop. Source: Apple WWDC 2000 Session 144 — Mac OS X: Application Packaging and Document Typing, 3:22

First, what is AppImage? It is a system for third-party application developers (think Ultimaker Cura) to release their software whenever they want, however they want, to Linux users without having to deal with different distributions, distribution policies, nor distribution infrastructure, without the need for maintainers, and without interfering what users may or may not have installed on their local machines. Conceptually, an AppImage is vaguely similar to a Mac .app bundle inside a .dmg disk image. The main difference is that applications stay inside the AppImage all the time and are not copied out, similar to how hpkg packages on Haiku get never installed in the traditional sense but mounted.

In the more than 10 years of existence, AppImage has gained quite some traction and popularity, with Linus Torvalds publicly approving it, and reputable projects (such as LibreOffice, Krita, Inkscape, Scribus, and ImageMagick) have been adopting it (mainly as an easy way to distribute continuous or nightly builds that do not interfere with whatever may or may not be installed on users’ systems). Linux desktop environments and distributions, however, are overwhelmingly still clinging to the traditional, centralistic, maintainer-based distribution model, and/or are pushing their own corporate, business, and/or engineering agendas with systems like Flatpak (Red Hat, Fedora, GNOME) and Snappy (Shuttleworth, Canonical, Ubuntu). To the point that it becomes comical.

How does it work?

Each AppImage is a combination of two parts: A tiny ELF runtime (called runtime.c) that is executed when the AppImage is double-clicked, followed by a SquashFS filesystem image

The squashfs filesystem image holds the payload application and everything it needs to run that cannot reasonably be assumed to be part of the default installation of every target system (Linux distribution) in a recent enough version in the default installation. It also holds metadata such as the application name, icons, MIME types and such

When it is executed by the user, the runtime uses FUSE and squashfuse to mount the filesystem image, then proceeds to launching a fixed entry point (called AppRun) inside the mounted AppImage. When that process exits, the filesystem image is unmounted and the runtime exits

Sounds easy enough, right?

This is what makes it cumbersome:

With different Linux distributions around, you can never know for sure what “reasonably can be assumed to be part of the default installation of every target system (Linux distribution) in a recent enough version in the default installation”. We have been working around this by compiling lists like the excludelist to determine what should not go into AppImage, and assume that we do need to bundle everything else. And even if this is generally working well, can still be still miss-and-hit sometimes. Hence we recommend AppImage creators to test AppImages on all target systems (distributions)
The payload application needs to be made relocatable in the filesystem. Unfortunately many Linux applications have absolute paths compiled in, e.g., to resources in /usr/share. Those need to be patched. Also, in order for the loader to find bundled libraries, either LD_LIBRARY_PATH needs to be exported, or the binaries’ rpath needs to be patched. The former has its downsides (which can be mitigated by complex workarounds) and the latter is just cumbersome
The biggest UX gripe that users have is that when they download an AppImage, they need to set the executable bit. Believe it or not, this is a big usability barrier for some users. Even for seasoned users, having to manually set the executable bit for each AppImage is cumbersome. Our workaround is to ask the user to install a small daemon on their system that constantly watches certain directories for AppImage files, and sets the executable bit on them. This is clearly not ideal because it does not work out of the box. Linux distributions are not shipping this daemon, hence the user still has a bad out-of-the-box experience
Linux users expect applications to show up in the launch menu and have an icon there. There is no way to tell the system “look, there is a new application here, do your thing”. Instead, as per the XDG specification you need to copy a desktop file to a certain location in /usr (system-wide) or $HOME (per-user). For icons, the XDG specification requires one to put graphics of certain sizes into certain locations into /usr or $HOME, and either actively run some desktop manager commands to refresh icon caches or hope that the desktop manager is smart enough to do so automatically. Similar for MIME types. Our workaround is that the daemon that sets AppImages executable checks whether the AppImage contains those files, and if it encounters them, copies them out of the AppImage and into the locations required by the XDG specification. When an AppImage is deleted or moved, the daemon is supposed to clean up again. Of course every desktop manager behaves slightly differently in which graphics files in which sizes it accepts in which locations, and how caches can be refreshed, making this a real pain. You can clearly already see that it’s a kludge
If all of this wasn’t enough, the AppImage still has no icon in the file manager. Since the Linux world so far has refused to adopt elficon (despite discussion and implementations), it is not possible to embed an icon right into an ELF executable. As a result, no application (AppImage or otherwise) has its own icon in the file manager, only in the start menu. As a workaround, we use thumbnails, a mechanism that originally was intended by desktop managers to have previews for e.g., graphics files, as their icons. Hence, the daemon that sets AppImages executable also acts as a “thumbnailer”, and generates and writes icon thumbnails to certain locations in /usr or $HOME. Also here, when an AppImage is deleted or moved, the daemon is supposed to clean up again. Of course every desktop manager behaves slightly differently in which graphics files in which sizes it accepts in which locations, and how caches can be refreshed, making this a real pain as well
If an error occurs (e.g., a library has not been bundled in the AppImage that is not part of the base system), then the application just fails to execute, and nothing tells the user in the GUI what is going on. We have started to work around this a bit by using desktop notifications, but it means that we’d have to catch errors from the command line, transform them into something the user can understand, and then display them as notifications on the desktop. Of course, every desktop environment handles those slightly differently
To this date I have not found a simple way to tell the system to always open 1.png with Krita and 2.png with GIMP

freedesktop.org is the host for XDG cross-desktop specifications used by GNOME, KDE, and Xfce

The freedesktop.org XDG cross-desktop specifications, and desktop managers’ implementations thereof, make it hard, if not impossible, to reach the level of elegance that is deeply woven into the Haiku desktop.

Just to give one example, it apparently didn’t cross the XDG authors’ mind that a user would have multiple versions of the same application around, since there is only one Firefox icon for the whole system, not per application version.

I have been wondering what Linux land could learn from Mac OS X to suck less at system integration. If you have some time and are into this kind of stuff, be sure to read it — Arno Gourdol, one of the original Mac OS X engineers, even chimed in:

We wanted the task of installing an application to be as simple as dragging an app icon from somewhere (a server, an external drive) onto your computer’s disk and be done with it. To do this, all the information that the system needed to know to handle the application was stored inside the app bundle, including its icons, its version information, the type of file it could handle, and the type of URL schemes it could handle. Then this information for ‘centralized’ in the Icon Services and Launch Services database. To maintain performance, applications were ‘discovered’ on a limited basis in a few ‘well known’ locations: the Applications system and user folders, and a few others, as well as automatically when the user navigated in the Finder to a directory containing an app. In practice, this worked very well.

Apple WWDC 2000 Session 144 — Mac OS X: Application Packaging and Document Typing. Source: Apple WWDC Videos

Nothing of this infrastructure seems to exist on the Linux Desktop, which is why we have been worknig around the strucutral limitations in the AppImage project.

Besides, the Desktop Linux platform is generally under-specified to the extent that many things that are trivially easy in a consistent full-stack system become frustratingly fragmented and complicated in Linux land. I have dedicated a whole talk to such Desktop Linux Platform Issues (and have been told, by the Linux developers in the knowledgeable audience, that this basically is how things are and will be for a long time to come).

My talk about 2018 Desktop Linux Platform Issues

Even Linus Torvalds has admitted that fragmentation is why Desktop Linux failed.

So refreshing to see Haiku!

With Haiku, everything becomes stunningly simple

While the naive approach to “porting” AppImage to Haiku would be to simply try to get its components, mainly the runtime and the daemon, to compile (which may even be possible!), it would not do Haiku any justice. Because actually, Haiku has solved most of those issues already, and in conceptually sound ways.

Haiku provides exactly the system infrastructure building blocks that I have been looking for on the Linux desktop for so long and could not believe they were not there. Namely:

Believe it or not, this is what many Linux users struggle with. Haiku just does it automagically!

In Haiku, ELF files that are missing the executable bit get this bit set automatically if double-clicked in the file manager
In Haiku, applications can embed resources like icons, which are shown in the file manager. No copying around of graphics files into defined icons directories, hence nothing to clean up when an application is deleted or moved
In Haiku, there is a database that connects applications with documents, and it is not necessary to copy any files around to make that happen
In Haiku, the lib/ directory next to an executable is automatically searched for libraries by default
In Haiku, there are not multiple distributions and desktop environments. So what works, works everywhere
In Haiku, there is no separate launcher that would be different from the Applications directory
In Haiku, applications normally do not have compiled-in absolute paths to their resources; in fact there are special functions in Haiku to find out locations at runtime
In Haiku, there is already the concept of compressed filesystem images: Every hpkg package is one. They get loop-mounted by the kernel
In Haiku, every file is opened with the application that has created it, unless the user explicitly specifies something else. How cool is that!

Two png files. Note the diferent icons to indicate that they will be opened by different applications when double-clicked. Also note the “Opens with:” dropdown menu whee the user can select a different application. Simple!

Looks like many of the kludges and workarounds we need for AppImage on Linux would not be needed on Haiku, which has simplicity and elegance woven into its fabric into how it handles most of what we need

Does Haiku need application bundles, after all?

Which brings me to the bigger question. If building a system like AppImage was feasible and by an order of magnitude easier on Haiku than on Linux, would it even be a worthwhile effort? Or has Haiku and its hpkg package management system factually removed the need for the concept of application bundles in passing? Well, to answer that, we need to look at the motivation why AppImages exist.

The user perspective

Let’s turn our attention to the end user:

As a user, I want to get an application without needing to ask the administrator (root) user to install it. — On Haiku, there is no administrator, the user is in full control, since it is a “personal” computer system! (It may, eventually, get some notion of multiuser — I hope they keep things simple.)
As a user, I want to run the latest and greatest versions of applications, without waiting for them to land in my Linux distribution (which more often than not is “never”, at least not without upgrading the whole operating system). — Haiku “solves” this by being a rolling-release system. Which means you can get the latest and greatest versions of applications, but also means that in order to do so, you need to constantly update the rest of the system as well, effectively making it a “moving target”
As a user, I want to have multiple versions of the same application alongside each other, since I can never know what is broken in the latest version, or since I am doing e.g., web development where I need to test my work under different browser versions. — Haiku solves the former but not the latter problem. Updates can be rolled back in time, but only for the system as a whole, it is not possible (at least to my knowledge) to run e.g, multiple versions of WebPositive or LibreOffice in parallel.

A Haiku developer comments:

Essentially the rationale is that this usecase is so rare, optimizing for it does not seem to make sense, and handling it as a special case in HaikuPorts seems more than acceptable.

As a user, I want to store applications wherever I like, not just on the boot disk. My disks are constantly running out of space, so I want to have an external drive or network share on which all the apps (in all versions I ever downloaded) reside. When I plug in this drive, I want to instantly be able to double-click and run those applications. — Haiku does store old versions of packages, but I don’t know how to move them to an external drive, and how to invoke the applications inside those packages from there

A Haiku developer comments:

It is technically possible to do this using the mount command already. If enough users want it, we can look at implementing it at the GUI level, indeed.

As a user, I don’t want millions of files scattered around in the filesystem which I cannot manage by hand in the file manager myself. I want one file per application that I can easily download, move around, and delete. — Haiku solves this somewhat through .hpkg packages, which bring down e.g., Python from thousands of files to one file. But if an application, e.g., Scribus, uses Python, then I have to deal at least with two files, namely, Scribus and Python. And I need to take care that I keep versions of both that can work with each other.

Many different versions of LibeOffice AppImages running alongside each other on Linux. Credits: @pdfkungfoo

The application developer perspective

Let’s now take the perspective of an application developer:

As a developer, I want to be in control over the whole user experience. I don’t want the operating system to dictate when and how I can release. — Haiku does allow developers to operate their own hpkg repositories, but this means that users have to manually configure them which makes them a bit of a “lesser” experience
As a developer, I have a download page on my website where I already provide an .exe for Windows, a .dmg for the Mac, and an .AppImage for Linux. Maybe I only want to offer the download behind a paywall, who knows. What am I supposed to put there for Haiku? — A .hpkg file that has dependencies only on packages in HaikuPorts
As a developer, I may need certain versions of things. For example, Krita is known to require a patched version or Qt that is exactly fine-tuned to a particular version of Krita, at least until the patches have landed upstream in Qt. — Haiku would probably allow me to bundle a private Qt in the application hpkg, although it would probably be frowned upon

A typical download page of an application. What to put there for Haiku?

Would it be a worthwhile addition to the Haiku desktop to have application bundles (be it in the form of application directories like AppDir or Apple-style .app bundles) and/or application images (be it in the form of greatly modified AppImages or Apple-style .dmg disk images)? Or would it just dilute the vision and add fragmentation, and hence complexity? I am torn: On the one hand, the beauty and elegance of Haiku comes from the fact that there is usually one way to do things, not many. On the other hand, most of the infrastructure for application directories and/or application bundles is already in place, so the system is screaming for the remaining few percent to also get into place.

Haiku developer mr. waddlesplash:

On Linux at least, they are more of technical solutions to systemic problems. Haiku has always preferred to just fix the systemic problem instead of the quick technical fix.

What do you think?

Before you answer…

Hold on, let’s do a quick reality check: In fact, application directories are already a thing in Haiku:

Application directories already existin Haiku, but are not really supported by the file manager so far

It’s just that they are not as well supported as they are as say, in the Macintosh Finder. How cool would it be if the QtCreator directory in the upper left hand corner had the “Qt Creator” name and icon, and would launch the application when double-clicked?

Some time ago, I had asked:

Are you sure that you will still be able to run your applications from today in 10 years time, when all the app stores and distribution repositories will long have forgotten about them and their dependencies? Are you sure you will still be able to access your work from today?

Does Haiku have a good answer to this question yet, or could application directories and application images help here? I think they could.

Haiku developer mr. waddlesplash:

Yes, we have an answer to the question: we will just keep those applications running for as long as is necessary until something else can suitably read their file-types, or provide 1:1 functionality. Our commitment to keeping BeOS R5 applications working on Haiku should be more than proof of this…

So true!

Where course of action should Haiku take?

I can imagine a peaceful co-existence of hpkg, application directories, and application images:

For system software .hpkg is used
For most regular applications (especially those the user wants to have on a rolling update schedule) .hpkg is used (80% of the cases)
Some of the applications installed through .hpkg would benefit from infrastructure for application directories (think Qt Creator); those would still be distributed as .hpkg files

Writes mr. waddlesplash:

If all you want to do is browse around in /system/apps , what we should do instead is make the Deskbar folders more manipulable by users, as /system/apps is not (unlike on macOS) intended for users to open and browse on a regular basis. So I think this is just a case where Haiku has a slightly different paradigm — but that should be acceptable.

For nightly/continuous/test builds, software the user wants to “freeze in time”, for private/in-house software, and for other special use cases like moving applications to external drives or network shares, Haiku gets the infrastructure to run application images (20% of the cases). Those application images could internally mount a set of hpkg files, run the application, and then unmount the .hpkg files again. (One could give this a decidedly Haiku spin: Possibly the file manager could put existing .hpkg files into application images automatically and on demand as the user does a drag-and drop of an application, e.,g., to a network share or external drive. On the reverse, the user could decide to install the .hpkg contents of an application image, at which point they would be treated and updated as if they were installed through HaikuDepot… let’s brainstorm.)

mr. waddlesplash:

Running applications off external drives or network shares does sound potentially valuable. Adding the ability to configure more pkgman “zones” may be a good feature indeed.

A system like this would leverage the advantages of hpkg, application directories, and application images. Individually, they are good, but combined, they will be unbeatable.

Conclusion

Haiku has infrastructure for providing a straightforward and elegant desktop user experience that is way and beyond over what Linux desktops commonly have to offer today. The .hpkg packaging system is one example, but other such elegant infrastructure is sprinkled across the whole system.

Nevertheless, I think Haiku could benefit a lot from proper support for application directories and application images.

How this could be best done is of course subject to discussion with people who know Haiku, its philosophies and its architecture way better than myself. I have been using it for less than a week, after all. Nevertheless, I hope this fresh outside perspective can be helpful for Haiku designers, architects, and developers. In any case, I am happy to help, if only as a sparring partner. I do have practical, hands-on experience with application directories and application bundles on Linux since over a decade, and would like to see them thrive on Haiku which is, in my opinion, ideally suited for the concept.

This is not to say that the potential solutions I have proposed are the only way to solve address the use cases I have described. If the Haiku team decides to find other— more elegant — solutions for the scenarios I have outlined, then I am all for it. In fact, I am already pondering an idea on how to make .hpkg even more awesome, without changing the way it works.

Turns out the Haiku team had already thought about bundles way back when they introduced package management. Unfortunately (I think) the idea was moved to “obsolete old ideas”. Maybe it’s time to revive https://dev.haiku-os.org/wiki/Obsolete/MovedToTree/PackageManagement/OldIdeas#Bundles?

What do YOU think about this topic? Leave your comments below!

This article is part of a series:

probono is the founder and lead developer of the AppImage project, the founder of the PureDarwin project, and a contributor to various open source projects. Screenshots were made on a Haiku system. Grateful acknowledgment is made to the developers in #haiku on irc.freenode.net, particularly mr. waddlesplash, @PulkoMandy, @diver, @begasus, and @humdinger.