This week started out with a search for some mechanism for building portable bundles for 64bit GNU/Linux systems. I looked at various bundler implementations, 0install, chakra, Alexander Larsson’s glick and glick2 to name a few of the implementations I found. Finally I found this gem called AppImageKit by Simon Peter (I refer to my own fork of his repository because it contains some patches I needed, however there is a pull request, which I think is the custom when doing things on github.com).
Before I continue, I should make clear what were my motivations and why I chose AppImageKit over any other existing options.
Why Bundled Applications ?
Bundled applications is a concept prominent in OSX and has been considered and reconsidered a lot by the general GNOME/GNU/Linux communities, I refer to Alex’s blog for some ideas on the concepts of bundling applications, you may also want to read this article by Lennart Poettering on the subject of Sandboxed Applications for GNOME (if you haven’t already seen his talk).
So first let me explain the motivations behind my hacking:
- I wanted to make bleeding edge versions of Glade available to a wider user base. While many serious developers will naturally have a copy of GTK+ master from git.gnome.org built locally (as naturally you target next generation software for the apps you create), we still have many users who are just not into creating a relocated build system and building everything themselves.
- Pretty much the same as the above point, I don’t want users to have to lag 3 years behind our development just because they decided to run a stable operating system (this should be a no brainer, really).
- Finally, I wanted a method of distributing portable binaries of proprietary applications, without the fuss of wallowing around in the muck of creating rpms and debian packages for every different system out there, I want to just create one bundle which will run pretty much everywhere with X11 and 64bit linux. Yes, time to throw eggs and tomatoes at me in public. Just because I’ve contributed lots and lots of my own time to work on free software for ‘free as in beer’, and have been a proponent of Free Software for my own idealistic reasons (I just believe that code developed in public is held to a higher standard of quality), doesn’t mean that all the software I write is going to be free software.
Why did I choose AppImageKit ?
If you’re already familiar with the scene of Application Bundles out there, then you’ve already probably guessed why I made my choice.
In my search for application bundling mechanisms out there, I found a few options (as I mention in the beginning of this post). I quickly realized that most of the projects I found were either targeting a specific OS/distribution (i.e. chakra) or at least required the user to install some kind of mechanism to run the bundle (i.e. 0install). While I really do respect and admire the work done by Alexander and the advocacy by Lennart, pushing for a new concept of packaging in GNU/Linux systems, my requirements were simply different (and perhaps once sandboxed applications for GNOME actually exist, I will be able to switch to that new mechanism).
My requirement is, again, to be able to download and run an application on your 64bit GNU/Linux platform, I don’t care if it’s Fedora, GNOME OS, Debian, Arch Linux, Ubuntu, Chakra, I just wanted it to run.
How does it work ?
After reading the first paragraphs of the AppImageKit documentation I was sold. This is a simple and nifty idea of simply creating a compressed ISO image with an ELF binary header, it uses libz to decompress itself and fuse to mount itself into a temporary directory, and then it just runs a script, letting you modify your environment and launch your App from within the unpacked bundle environment (actually the AppRunScript.sh is my own addition to AppImageKit). It also uses libglib for some obscure reason, so the best practice is to build the bundling mechanism with the oldest version of libglib possible (as this will become a base runtime requirement for the user).
So basically the requirements are:
- C runtime libraries
- libfuse, with a fuse capable kernel
- libz
- libglib (as old as possible)
After that, the system requirements depend entirely on what you put in the bundle.
Please test my Glade bundle !
After 2 days of hacking I was finally able to get a GTK+ application running (Glade) with the Adwaita theme fully functional, and self contained. This involved much ldd and strace in the bundle environment, ensuring that the runtime doesn’t touch any system libraries or auxiliary data, the discovery of this nifty chrpath tool, and much grepping through various sources and figuring out how to properly relocate these modules with environment variables. I had to strangle Pango, until it finally yielded, after I thoroughly deformed pango’s face with this downstream patch. I also needed to rediscover hicolor-icon-theme’s location on fdo, since everything breaks down without an index.theme file in the right place (one of the various things jhbuild forgets to do properly).
You might be interested to peek at the README which is a thorough breakdown on how to create portable bundles of Glade in the future, the application startup script is also of particular interest as it shows what environment variables you really need to relocate GTK+ apps.
Finally, I was able to produce this bundle.
In addition to the AppImageKit system requirements listed above, we require:
- X11 libraries
- fontconfig
- freetype
(Technically these could also be included in the bundle, it would require that we install fonts as well, and I just wanted to avoid that telling myself that most GNU/Linux distributions have this already).
The above bundle should be able to ‘just run’, just download it and run it without any special privileges… PLEASE !
We will greatly appreciate any testing on various 64bit GNU/Linux platforms, after some testing we will start to advertise downloadable bundles of bleeding edge Glade for any users of stable GNU/Linux distributions.
And of course, Enjoy !
Works great on a Ubuntu 12.04! Also, thanks for the detailed description of how to get things running with AppImageKit. This could come in handy for some projects I work on.
works here as well (arch linux 64-bit, GNOME 3.10), but had to give it permission to execute itself (don’t know if that counts as a ‘special’ permission?). It didn’t work out-of-box (“there is no application installed for “executable” files”). but nice job 🙂
not so great news from debian wheezy:
./glade: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14′ not found (required by ./glade)
i have 2.13
http://packages.debian.org/wheezy/libc6-amd64
Yes, it works, but with some caveats:
boud@linux-ixka:~> ./glade
bash: ./glade: Permission denied
boud@linux-ixka:~> chmod a+x glade
boud@linux-ixka:~> ./glade
GLib-GIO-Message: Using the ‘memory’ GSettings backend. Your settings will not be saved or shared with other applications.
GladeUI-Message: No DevHelp installed on your system, devhelp feature will be disabled.
(glade:22006): GladeUI-WARNING **: Failed to open catalog directory ‘/usr/share/glade/catalogs’: Error opening directory ‘/usr/share/glade/catalogs’: No such file or directory
GladeUI-Message: Glade needs artwork; a default icon will be used for the following classes:
GtkApplicationWindow needs an icon named ‘widget-gtk-applicationwindow’
This is on OpenSUSE 13.1, with KDE running but all of Gnome3 installed. I’ll also check on another system.
I’m rather interested in doing distribution-independent, easily runnable packages of Krita myself. One thing though that’s not optional, is saving settings to the user’s home directory. Did you intentionally not enable that or is that a limitation of AppImageKit?
@Luis: It’s not the worst of news
I’m sure it can work there, however I really need an older system to build the bundler with.
It’s possible however that I can trick the linker into requiring older versions and staying ABI compatible, I’ll have to look into this.
@Boudewijn: I’ve been hearing mixed reports about it being executable, this could simply be an attribute of how your browser downloads files (but this is of minor importance, of course).
Regarding the other issues you mention… DevHelp right now is try and miss, it could work, or DevHelp could crash when launched from the internal bundled environment.
My thoughts were to just disable DevHelp from the bundle, but I’m considering just leaving it this way, as on some systems it works (and if Glade detects it did not work at startup time, Glade *should* not show the documentation options in it’s UI, however this might be broken).
We intentionally do not interact with the system installed GSettings, our internal bundled environment might make use of the memory backend internally, mostly for GTK+ purposes (I think the file chooser for instance accesses the GSettings in the bundle).
The intention is to restrict any interaction with the system and stay self contained.
However, as Glade itself uses GKeyFile to store settings in XDG_CONFIG_HOME,
Glade itself does indeed access the same key file that an installed version of Glade
would.
Note that the AppImageKit itself deserves experimentation, if you really want to use GSettings in another way, you can try it. It might mean building in a specific GSettings backend (preferably without D-Bus interaction) into the bundle itself, you would have to try it out.
Remember of course, the more you try to interact with the user’s system, the less portable is your bundle.
What does concern me however is the message:
“Error opening directory ‘/usr/share/glade/catalogs”
Glade should not be trying that, however it might just be a problem trying to read the system Glade installation catalog directory (not a catalog itself, I’ve disabled loading of various user catalogs which Glade might otherwise try to load from the system), as you mention ‘it works’, I suppose this is just an error message which I can easily fix.
Cheers, and thanks for trying out this bundle !
[nick@zeus Downloads]$ chmod +x glade
[nick@zeus Downloads]$ ./glade
bash: ./glade: cannot execute binary file
[nick@zeus Downloads]$ uname -a
Linux zeus 3.11.9-200.fc19.i686 #1 SMP Wed Nov 20 21:41:11 UTC 2013 i686 i686 i386 GNU/Linux
Looks like it’s a dud on Fedora 19 🙁
@Nick: Thanks for trying it out
“bash: ./glade: cannot execute binary file”
That’s a bit obscure 😉
Can I ask what did ldd have to say about the binary ?
I suspect that, maybe you do not have libfuse ?
Thanks Tristan, seems to work well on my 64bits Mageia 3 system. A few errors, though:
================================
[luis@gontran Téléchargements]$ ./glade
GLib-GIO-Message: Using the ‘memory’ GSettings backend. Your settings will not be saved or shared with other applications.
devhelp: /tmp/.mount_teKGZ2/usr/lib/libxml2.so.2: version `LIBXML2_2.9.0′ not found (required by /lib64/libxslt.so.1)
(glade:9020): GladeUI-WARNING **: devhelp had bad return code: ‘256’
GladeUI-Message: Glade needs artwork; a default icon will be used for the following classes:
GtkApplicationWindow needs an icon named ‘widget-gtk-applicationwindow’
=================================
For the libxml part, I have 2.9.0 installed:
[luis@gontran Téléchargements]$ rpm -ql lib64xml2_2
/usr/lib64/libxml2.so.2
/usr/lib64/libxml2.so.2.9.0
liberforce: This is sort of expected, see the comment I made previously to Boudewijn regarding DevHelp.
Currently we make an attempt to integrate with your system’s DevHelp, and if that doesnt work, well no harm done, just keep trucking along happily without DevHelp support.
The cleaner alternative(s), would be to:
a.) Just disable DevHelp in the bundle
b.) Include DevHelp in the bundle (making it HUGE and including webkit)
Ubuntu 12.04 LTS. Works perfect.
Works great with OpenSuse 13.1.
Thank you very much for your work! I wished There would be more development in that direction. Maybe in Gnome OS…
Hi,
works without problems on Arch Linux, thanks!
This solution should be great for Humble bundle games for example 🙂
It work in my ArchLinux x86_64. Nice job. For 3rd party apps it is very useful, use more than one version of an app too and do not touch on my system!
Gnome projecto told very mouch about glick2 and bundled apps for 3rds but recently they did metion nothing about that. 🙁
Glad to see the Glade AppImage 🙂 The provided AppImage is an “ELF 64-bit LSB executable, x86-64” so it does not run on some (older) computers/OSes. Maybe you can provide a 32-bit version as well.
@Nick G – it’s for 64-bit only so won’t run on an i686 Fedora
Had to do a little dance to “allow executing file as a program” on Fedora 20, but apart from that, worked great! Thanks Tristan!
Also works under gnome-continuous
No luck on Fedora 17 x86_64:
$ ./glade
/tmp/.mount_uDTwGP/AppRunScript.sh: line 30: /tmp/.mount_uDTwGP/usr/bin/glade: No such file or directory
Please let me know if I can provide you with any useful debugging information.
@Andrea: Wow, this one is really surprising (doesn’t seem to be in the realm of possible).
We do rely on /bin/sh to run the script, but the script was certainly found… the file is most certainly there in the bundle (or it would have equally failed for everyone).
Which leads me to wonder if this is an instability of fuse, perhaps ?
I am currently sorting out the “too new libc requirement” problem, and I have a solution I like but want to create a better build script before I try to rebuild everything, possibly a stand alone jhbuild moduleset, to ease the process of these builds… however I’ll post a better debug bundle for you first thing tomorrow (as it’s 2:30am here I wont do it now).
I’ll post a test bundle which tries to launch Glade and then runs ‘/bin/bash’ (this is what I’ve been doing to test), this way you can see what’s going on much more easily from inside the environment which launches Glade.
Please make gedit 3.10 and nautilus 3.10 bundles!
I need gedit 3.10 for regular expressions and nautilus for ‘open file location’ in search
Thanks.
@Asif: I think this will be very possible for gedit, I wouldn’t bet on nautilus though 😉
For nautilus to work well, I would expect it to need to interact with the system, this means it would (probably) require some specific versions of D-Bus APIs available on your session bus (for an example).
It’s possible I’m wrong, but I doubt this ‘stand alone bundle’ will work for apps like nautilus, who’s purpose is to really integrate with your desktop.
@Tristan
thank you, gedit would be great, also it can help as a guide package to other gnome apps into bundles.
We, really need good and simple documentation/howto.
@Asif: Yeah, Glade will have a build/ subdir dedicated to this guide/template
I’ve spent the day changing my original LFS style spike… into a custom jhbuildrc and moduleset file.
With this, it should be possible to just copy what we did in Glade and pretty quickly get a bundle working for any GTK+ app (of course more complex apps with more dependencies will be more tricky, but I think this is the best I can make it)…
I should be committing something tomorrow with an update.
@Andrea: I’ve updated the test bundle at the same link, I still have no idea what’s causing this “No such file or directory” issue for you, but the new bundle is much more forgiving regarding libc versions.
Also, it can now be run in test mode as such:
APP_IMAGE_TEST=1 ./glade
This will give you a shell, from inside the shell, you can launch glade with:
./bin/glade
I’d be interested to know if the new bundle still doesnt work for you with Fedora 17, and if it does not, I’d be interested to know what happens if you try to launch it with the test environment as described above.
Some points of interest besides trying to launch might be:
ldd bin/glade
LD_DEBUG=all ./bin/glade
Or in your case, I would also be interested in this:
find . > ~/bundle-files.list (and paste that list for me somewhere)
The updated bundle now works for me, where it previously didn’t (Fedora 19). Very nice!
One issue with the bundled version seems to be that it no longer loads glade catalogs from the usual location, and setting up a catalog path doesn’t seem to do anything.
@Kai: This is intentional.
The warning message about not able to open the catalog message on your console is not intentional, though, just some cleanup I still have to do.
We explicitly ensure that we don’t load anything from the random user’s linux distro. We could eventually allow this with bundles so long as we’re only loading XML, however there is a real danger that most catalogs point to libraries on the random linux system.
Glade bundles loading modules from your system can only lead to tears, unfortunately.
This reminds me of all the shenanigans with http://www.pixelbeat.org/programming/linux_binary_compatibility.html
On Fedora 15 it loads by crashes on menu click with:
“/home/tristan/Glade/Install/lib/libgdk-3.so.0: undefined symbol: _XGetRequest”
BTW when providing binaries it’s really advisable to sign them,
so they can be verified. I usually generate a detached sig with:
gpg –sign -ab -o glade.sig glade
@Pádraig: Indeed nice link !
Unfortunately it’s necessary to learn about .symver, again, and again, and again, generation after generation, because believe it or not, gcc until this day STILL does not provide any option to –select-abi-target=glibc,2.5.
Thanks for trying the binary, it looks like it actually would be worth while going the last mile and providing Xlibs,fontconfig and freetype inside the bundle (I’m not looking forward to this, as fontconfig get’s particularly cranky when you try to tell it that it’s wrong about it’s –prefix).
I’ll consider what you said about signing, I’m thinking of publishing these on http://ftp.gnome.org in a similar way that we would normally roll win32 binaries, so I’ll certainly sign those.