This is a follow up post on the one I made earlier this week on Glade application bundles.
What I had hoped to be a simple bundling experience turned out to actually take me all week to complete, but hey, you get what you pay for. And I’m very happy with the outcome so it was well worth the effort.
The new test bundle can be found here, and I’m very proud to announce that we require exactly the 2.7 ABI form glibc.
We still require fontconfig/freetype/Xlib libraries to be present on the target host to run what is inside the bundle, and we still require libz and a moderately old version of libglib to actually unpack the bundle.
For the AppImageKit glib & zlib dependencies, I plan to make those static dependencies so that they will not even be required, but first I must do battle with CMake to make that happen.
How can I use this to bundle my GTK+ application ?
This is a question I’ve been expecting to hear (and have been hearing). As the last iteration was only a spike, it involved a very ‘LFS’ technique of manually preparing the bundle, but that has been fixed this week and now you can use the bundling mechanism with a collection of customized jhbuild scripts.
Here is an outline of what we have in our build/linux/ directory, which can serve as a good template / starting point for others to bundle their application:
- README: A step by step instructions for generating the bundle
- jhbuildrc: A custom jhbuildrc for building the bundle
- AppRun: A script to run your GTK+ app from inside the bundled environment
- PrepareAppDir.sh: A script to post-process the built stack including GTK+, it’s dependencies and your app
- LibcWrapGenerator.vala: A program for generating libcwrap.h
- modulesets/bundle.modules: A self-contained moduleset for building the stack
- modulesets/patches/: A few downstream patches needed to build/run the bundle
- triggers/: Some custom jhbuild post-installation triggers
People are welcome to copy this build/linux subdirectory into their project and try to build their own GTK+ based application bundles.
Different applications will have different requirements, you will certainly need to modify AppRun and bundle.modules.
Hand to hand combat with glibc
Most of my work this week has revolved around doing hand to hand combat with glibc. I do feel a bit resentful that gcc does not provide any -target-abi-version=glibc,2.7 sort of command line option, as this work would be much better suited to the actual compiler and linker, however with libcwrap.h and it’s generator, I’ve emerged victorious from this death match.
The technique employed to get the whole stack to depend on glibc 2.7 ABI involves using the .symver directive like so:
__asm__(".symver memcpy, memcpy@GLIBC_2.2.5");
The symbol memcpy() changed in libc’s 2.14 ABI to behave more like memmove(), however linking to the new symbol causes your application to require libc 2.14 ABI, which of course we don’t want. The above .symver directive ensures that you will link to the version of memcpy() which is provided by the glibc 2.2.5 ABI.
However the problem is more intense than just this, there are many symbols in libc with various versions and what we want is to be able to simply choose a target version. The LibcWrapGenerator.vala program takes care of this.
Running the generator on a given platform will call objdump on your system libc libraries and inspect which symbols exist, which of them can be safely accessed without any .symver directive, which symbols can be redirected to their most recent version before your arbitrary target ABI version and finally, of course, which ones simply are not available, for instance the fallocate symbol:
__asm__(".symver fallocate, fallocate@GLIBC_DONT_USE_THIS_VERSION_2.10");
The fallocate symbol was actually initially introduced in the 2.10 ABI, and simply cannot be used when targeting the glibc 2.7 ABI.
The nice thing about the above directive, is that it will cause an informative linker error when you try to link any program using fallocate() against glibc. Some of my patches work around these problems (for instance cairo uses longjump() in it’s test cases, so my patch simply disables these test cases when building for the bundle), in other cases this is totally transparent. For the case of fallocate above; libglib will try to use it if it’s available, but with the forced include of the generated “libcwrap.h” file, glib simply fails to find fallocate at configure time and happily functions without it.
This technique was borrowed from the now defunct autopackage project which used a bash script with awk to generate the header file in question, however this script stopped working for glibc runtimes >= 2.10. A newer port of that old script to vala was created by Jan Niklas Hasse in 2011 which was a good starting point, but still failed to generate the correct output. Today I put my (non-existent) vala skills to the test and now have a working version of this nifty header generator, salvaged from the autopackage wreckage.
More Testing !
I will of course appreciate any more testing of the new bundle, for any systems with the glibc 2.7 ABI or later. We did prove that the bundle works properly for a CentOS 6 system which is already a good start.
I would be interested however to know if other people can build the bundle following my README file in our build/linux directory, specifically I’m looking for cases where you have a very new glibc ABI, generate a new libcwrap.h, and let’s see if the bundling process still works, and also let’s see if we can run the bundled glade on very old systems, generated from very new systems.
That’s all, I think I’ve covered all the ground I intended to in this post… so please try this at home ! 🙂
And enjoy bundling your own GTK+ applications 🙂
Tested it on arch linux and it seems to work fine except for these messages in the commandline:
“(glade:14616): GladeUI-WARNING **: Failed to open catalog directory ‘/home/tristan/AppImages/Install/share/glade/catalogs’: Error opening directory ‘/home/tristan/AppImages/Install/share/glade/catalogs’: No such file or directory
GladeUI-Message: Glade needs artwork; a default icon will be used for the following classes:
GtkApplicationWindow needs an icon named ‘widget-gtk-applicationwindow'”
🙂
@Bastian: Thanks ! This testing is really appreciated.
Is there anything you can tell me more about ‘arch linux’ ? I.e. is this a bleeding edge system or is it LTS ?
Also, pay no mind to the catalog related warning, I have to fix that warning in Glade and have been lazy about it (will do before publishing these bundles in any official location).
on fedora 19 x86_64:
http://pastebin.com/raw.php?i=bSbm1yCi
——partial—-
(devhelp:16871): Gtk-WARNING **: Theme parsing error: gnome-applications.css:488:25: Not a valid image
(devhelp:16871): Gtk-WARNING **: Theme parsing error: gnome-applications.css:504:25: Not a valid image
(glade:16866): GladeUI-WARNING **: Failed to open catalog directory ‘/home/tristan/AppImages/Install/share/glade/catalogs’: Error opening directory ‘/home/tristan/AppImages/Install/share/glade/catalogs’: No such file or directory
GladeUI-Message: Glade needs artwork; a default icon will be used for the following classes:
GtkApplicationWindow needs an icon named ‘widget-gtk-applicationwindow’
@Asif: Interesting, I see that devhelp is spewing a lot of warnings if launched by Glade from the bundle.
Currently I don’t think these warnings are all that bad, but I think it’s possible for me to play a better trick now that I think of it.
In the AppRun script we setup the internal environment to run Glade, but perhaps we should save the entire environment vector and somehow give that whole thing to Glade, Glade can then use the original (saved) environment for the purpose of launching DevHelp…
I’ll have to look deeper into that, but I don’t see these warnings as a blocker for shipping the Glade bundle (as DevHelp will either work or it wont, it doesnt strike me as all that bad).
AppImageKit does not compile here (fedora 20) when using the libcwrap trick. I am getting:
MakeFiles/runtime.dir/runtime.c.o: In function `md5_buffer’:
/home/thiblahute/cerbero/sources/linux_x86_64/appimagekit-0.1.0/md5.c:178: undefined reference to `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
libisofs.a(isofs.c.o): In function `isofs_parse_zisofs_header’:
/home/thiblahute/cerbero/sources/linux_x86_64/appimagekit-0.1.0/isofs.c:728: undefined reference to `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
libisofs.a(isofs.c.o): In function `isofs_real_readdir’:
/home/thiblahute/cerbero/sources/linux_x86_64/appimagekit-0.1.0/isofs.c:1502: undefined reference to `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
libisofs.a(isofs.c.o): In function `isofs_real_read’:
/home/thiblahute/cerbero/sources/linux_x86_64/appimagekit-0.1.0/isofs.c:1799: undefined reference to `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
libisofs.a(isofs.c.o): In function `isofs_real_read_zf’:
/home/thiblahute/cerbero/sources/linux_x86_64/appimagekit-0.1.0/isofs.c:1716: undefined reference to `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
/usr/bin/ld: runtime: No symbol version section for versioned symbol `memcpy@GLIBC_DONT_USE_THIS_VERSION_2.14′
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [runtime] Error 1
make[1]: *** [CMakeFiles/runtime.dir/all] Error 2
Any idea?
@thiblault: Ah, this is exactly the type of problem I’m looking to sort out.
First question:, how did you invoke LibcWrapGenerator ?
Particularly, what glibc target did you choose ?
The memcpy function should not fall under the DONT_USE_THIS_VERSION category but rather be directed to 2.2.5, so if you are targeting any libc version > 2.2.5 then this shouldn’t happen (but as I only have my own glibc to test against, I can’t be sure yet that it’s parsing objdump output properly).
Can you try this please for me ?
export LIBCWRAP_DEBUG=collect:filter
./LibcWrapGenerator -l /directory/containing/libc/ -o libcwrap.h -t 2.7 > debug.log 2>&1
And paste the log file in some pastebin that I can look at ?
Perhaps also paste the libcwrap.h it generated too ?
@thiblault: Actually, if you could also run: objdump -T /path/to/libc.so.6 and send me the output of that, that would be also very helpful.
What I suspect went wrong, of course, is the parsing of objdump output, as I’ve encountered 2 versions of this which did not work already, it’s certainly difficult to get right :-/
Hopefully with your objdump output I can see the reason why it did not work for you.
@tvb: I am targeting the glibc 2.7 too. I uploaded all the wanted files at: http://people.collabora.co.uk/~tsaunier/LibcWrapGenerator.zip
Wondering if it would not make sense to actually share a valid header file around for a particular target?
@thiblault: Thanks for the upload !
A header file can be shared but it’s delicate that way.
On my system I generate the header file based on the glibc 2.15 ABI that I have available.
This header file is safe to use when compiling on any system that has glibc <= 2.15
So if the header is generated against a brand new glibc every time glibc makes a release, then it would be safe.
This is why I think having a stable generator would be best (allowing one to have flexibility of the target glibc, and not risking the header file being used against a newer glibc which might contain new symbols that the header doesnt know about).
@thibault: Good news I think.
It’s a bit late for me to be debugging this tonight, but from a quick look at the log it looks like I messed up something in the vala code which should be easy to fix.
The good news particularly is this line from the debug.log:
Selected symbol ‘memcpy’ version ‘2.2.5’ from objdump line 0000003ba4689400 g iD .text 0000000000000047 (GLIBC_2.2.5) memcpy
Which means it’s parsed properly from the objdump output but the vala logic for some reason is broken, I’m sure there is a palm face connection waiting for me tomorrow 😉
@tvb OK, I got your point about sharing the headers, you are right.
Trying to compile libvorbis, I am getting the follwowing issue:
/usr/bin/ld: .libs/libvorbis.so.0.4.6: No symbol version section for versioned symbol `__exp_finite@GLIBC_DONT_USE_THIS_VERSION_2.15′
any tip to investigate that?
@thiblault: Ok thanks so much for testing my vala code 🙂
I found the bug, which was indeed quite easy to fix
I’ve pushed it to both locations I have control over:
https://github.com/tristanvb/AppImageKit/blob/libc-wrap-generator/LibcWrapGenerator/LibcWrapGenerator.vala
https://git.gnome.org/browse/glade/tree/build/linux/LibcWrapGenerator.vala
And setup a pull request to get it merged into upstream AppImageKit (I will soon remove it from the Glade repository, but I want to fix AppImageKit’s build scripts so that it builds itself using the generator automatically, first).
As per your question regarding __exp_finite().
This function is an optimization for glibc to make an assumption that floating point numbers are finite.
You should be able to work around this in libvorbis’s Makefiles, my guess is that
the libvorbis pass -ffast-math at compile time, this will in turn enable -ffinite-math-only.
The -ffinite-math-only is what is causing exp() to be redirected to the __exp_finite() symbol.
So you might try either removing -ffast-math, or adding -fno-finite-math-only after -ffast-math.
@tvb that work just fine thanks a lot 🙂 I am now having an issue with cdparanoia that uses clock_gettime and am getting
/usr/bin/ld: libcdda_interface.so.0.10.2: No symbol version section for versioned symbol `clock_gettime@GLIBC_DONT_USE_THIS_VERSION_2.17′
I tried passing -lrt but it does not change anything, any idea about how to work around that?
(btw I am bundling pitivi using cerbero, adding a packager for it in there)
@thiblault: Ok this actually looks like another bug in how we are parsing objdump :-/
On my system, I have my system libraries in /lib/x86_64-linux-gnu/
I don’t have such a new version of glibc (2.15 here), so I don’t have the new version of clock_gettime()… but I find clock_gettime() is available in:
/lib/x86_64-linux-gnu/librt-2.15.so
Running the LibcWrapGenerator on my platform finds the symbol clock_gettime() 2.2.5 without any problem, and I find that looking at your debug output which you sent me, the last buggy generator did also find clock_gettime() 2.2.5 and the new 2.17 version.
The fix for the LibcWrapGenerator.vala I sent should have fixed that, did you properly regenerate the header file with the new generator ?
Fetch a copy of the generator here:
https://git.gnome.org/browse/glade/tree/build/linux/LibcWrapGenerator.vala
Cheers
I had not rerun with your new version, I now did and am getting:
/usr/bin/ld: .libs/libgstaudiofx.so: No symbol version section for versioned symbol `memcpy@GLIBC_2.0`
in libcwrap.h I have __asm__(“.symver memcpy, memcpy@GLIBC_2.0”); instead of __asm__(“.symver memcpy, memcpy@GLIBC_2.2.5”);
and the same issue is happening with clock_gettime, in libcwrap.h I have __asm__(“.symver clock_gettime, clock_gettime@GLIBC_2.2”);
instead of __asm__(“.symver clock_gettime, clock_gettime@GLIBC_2.2.5”);
I also have the same issue with clock_getres@GLIBC_2.2 instead of clock_getres@GLIBC_2.2.5
Setting those correctly by hand fixe my issues.
Now compiling cogl I am getting:
/usr/bin/ld: .libs/cogl-winsys-glx.o(.text+0x17db): unresolvable R_X86_64_PLT32 relocation against symbol `clock_gettime@@GLIBC_2.17′
-lrt flag is passed so I am not sure how that can be solved, any idea?
Btw, would there be a better place to discuss that than here? 🙂
@thiblault: This is very confusing.
I read back through the original log you sent me, and the generator never encountered any clock_gettime 2.2, nor any memcpy 2.0 in the whole log.
Would you send me a new log again with all of the debug enabled ?
That would be with this environment:
LIBCWRAP_DEBUG=collect:filter:version-parse:version-compare
Also, the logic for selecting the right symbol versions starts at this line:
https://git.gnome.org/browse/glade/tree/build/linux/LibcWrapGenerator.vala#n243
I know the parser is extracting symbols and versions properly, and already changed the above logic once, do you spot a problem in there that I missed ?
@thiblault: you’ll probably catch my messages to you on #gnome-hackers… which is probably the best place.
But since you only appear online after 1:00am, I don’t think irc will work well for our timezones :-/
We can do this also by email, perhaps on: glade-devel at lists.ximian.com, or contact me directly at: tvb at gnome.org
@tvb, nevermind about the symbol versioning mixup, this is my fault I generated inside /usr/lib/ instead of lib64/, sorry about that.
This is awesome. After some months of apparent decay of the AppImageKit project, it seems to be twitching again!
You’ve got a very interesting take on the glibc issue, thanks a lot for the read.
The method I use to avoid glibc version dependencies on PortableLinuxGames.org is more like brute-force. I pack *everything*, including glibc (and I’m using Arch so it’s a pretty bleeding edge 2.18 glibc) and the linked (ld-linux.so.2), but it seems to work pretty well, even on “old” distros like Ubuntu 10.04. I borrowed this method from the CDE project (http://www.pgbovine.net/cde.html), and it seems to yeld pretty good results, although I don’t really have time to test all my packages in all the systems I should, so your mileage may vary wildly. And of course this method is not feasible to build AppImage the runnable itself.
I’ve recently started testing my packages on TinyCoreLinux, and stumbled upon the glib dependency issue. Have you made any progress on compiling it statically, or even removing the dependency from isofs.c? I was going to start trying to at least battle with CMake.
Btw, I’m sorry I couldn’t test your AppImage, I’m on a 32bit system. All my AppImages are 32bit, and I have started creating a small script to setup multilib for libfuse, at least on some distros. I think it’s more maintainable than packaging both 32bit and 64bit versions of each AppImage.
Woot. I installed glibc2-static from AUR, tweaked a bit the CMakeList, and got a statically linked runtime.
-rwxr-xr-x 1 raziel raziel 28K Dec 21 22:51 runtime-dynamic
-rwxr-xr-x 1 raziel raziel 328K Dec 21 22:52 runtime-static-glib
-rwxr-xr-x 1 raziel raziel 360K Dec 21 22:53 runtime-static-glib-zlib
-rwxr-xr-x 1 raziel raziel 59K Dec 21 22:53 runtime-static-zlib
Patch: http://pastebin.com/GJJ3qjxM
@Ismael: Interesting stuff indeed.
I may look into this, however parts of this are problematic for me.
First, I’m surprised that you got CMake to work this way, I did try that, and it was unable to find libglib-2.0.a in my system directory here:
/usr/lib/x86_64-linux-gnu/libglib-2.0.a
(I struggled for 4 or 5 hours before giving up on this).
I’m interested in making the bundler optionally build the LibcWrapGenerator.h, locate the system libc directory, generate the file, and build the bundle extractor with a low glibc dependency (if vala and libgee-0.8 are not found, it should simply warn you at configure time that the bundle will not require a low glibc dependency).
Do you think this is possible using CMake ?
More interesting is your technique of bundling glibc itself, this worries me because it means I will unconditionally need to build Xlib/freetype/fontconfig into the bundle, but as this technique I borrowed from autopackage is a bit unweildly for some systems (thiblault is reporting a really mysterious linker error for the clock_gettime() symbol), I may be willing to switch.
When you bundle for PortableLinuxGames.org, do you setup fontconfig inside your bundles ? do you install fonts ?
I would be interested to see the postprocessing you do on the output of ‘fc-cache’ to get the whole thing relocatable. Do you perform a ‘chroot’ in your script instead ? (I definitely want to avoid chroot, as applications need to have access at least to the user’s HOME dir, to read and write files and perhaps save some configuration data).
I risk being unresponsive for the next day or two (it’s moving time), but really would appreciate any added insight you may have on the subject.
If I can add glibc/Xlibs/freetype/fontconfig to the bundle and drop the wrapper script it may be a lot of work to get this automated in my build scripts, but it sounds like I would only need to do it once, and perhaps relying on the kernel system call interface for ABI compatibility will be more stable than tricking glibc into backwards compatibility.
Regarding libglib-2.0.a, it was not present on my Arch out of the box, but I installed the package glib2-static, which created /usr/lib/libglib-2.0.a, and then it worked flawlessly. If yours is in /usr/lib/x86_64-linux-gnu/ and it’s not being found, maybe there’s some PATH that needs to be updated.
Regarding making CMake detect vala, I don’t really know if it’s possible, but if the CMake can build an AppImage I’m pretty sure it can perform this kind of check as well, even if it’s done calling an external script.
Regarding fontconfig and freetype, I don’t setup anything, nor include any fonts; but everything works out of the box. Maybe it’s because I’m packing games, which are usually build using SDL/OpenGL, and usually with custom fonts, so system fonts are more likely to be ignored than in a GTK application?
I never use chroot; usually what I do when I’m packing a binary from my distro’s package manager, is to patch the application binaries and libs to turn for example “/usr/share” into “././share”. If the application doesn’t use ~/.app or ~/.config/, and needs to write in this readonly data folder, I use unionfs to redirect writes to ~/.local/. But no chroot nor fakechroot is ever used.
If you really need chroot, you can do what the previous incarnation of AppImage did, although it may get tricky. klik (https://code.google.com/p/klikclient/) used fakechroot together with unionfs, so that the application sees a different root without requiring root permissions (LD_PRELOAD trick), and the user’s home directory is overlaid on top of that chroot, with read and write permissions.
If you’re having trouble tricking glibc into requiring an older ABI, I think the sanest solution for the AppImage runtime would be to setup a lightweight virtual machine, or even a chroot, with a really old distro, and compile the runtime there, which I think it’s what probono does. For the application inside the package, I’d try including glibc, ld-linux.so.2, and all the font dependencies. It should be easy to automate for all the applications you build.
By the way, CDE is not restricted to games, and I’ve seen it working across very different distros with pygtk (I think) packages, support fonts and whatnot, so you may want to take a peek at that.
@Ismael: Requiring developers to produce an old/outdated environment for building with a low libc requirement is EXACTLY the answer which I will not accept.
I have to debug this further with thiblault, I have not encountered the linker error in question myself but would like to get to the bottom of that, rather than giving up.
Why ? Because I expect to be building bundles from scratch very often, I want this to gain traction, I want to be able to use it in a straight forward way.
I did get a chance to look at the papers/documentation on the concept of CDE, and while I admit it’s definitely a ‘neat trick’, it’s far worse of a hack than simply redirecting some symbols to older libc versions (and avoiding usage of new symbols which are only
introduced in new versions). I’m also not a fan of modifying binaries to replace ‘/usr’ with ‘././’, again, it’s a really neat trick when you have a proprietary binary and want to bundle that, and have no choice but to work with the binary, other than that it’s just a dangerous hack.
If thiblault’s linker error is really found to be not fixable, I would rather explore building a relocated glibc into the bundle, and packaging Xlib/expat/fontconfig/freetype into the bundle itself (this way I would only require ABI stability at the kernel system call level, which is probably arguably more stable).
Alternatively, we could explore the option of installing a non-primary glibc by building an old version of glibc, linking against that glibc but NOT including it in the bundle. This would be a safer and more pragmatic approach than the one I’m using now, but again I would rather not force that to be a part of the bundling process, as building a non-primary glibc and properly linking it might prove more difficult.
Disregarding bundles, compiling your application in the oldest system you want to support is pretty much standard nowadays for everyone that doesn’t stick to “compile one package on each version of each distro we want to support, and fuck the rest” nosense. But I agree it’s suboptimal, and any improvement will be a huge contribution in the state of Linux toolchains, so yours is a very promising development 🙂
Now, regarding bundles:
CDE only targets packaging pre-built binaries, so they need to perform an awful lot of tricks to make them work, but there’s still a lot to be learnt from some of them.
If I understood correctly, you only target binaries compiled from source, so you can start clean from scratch.
I’m in between, so I use CDE’s dirty tricks to package apps already available in my distro because it’s a quick way to gain traction among users, but I’m as interested as you are in getting this to gain traction among developers.
Then, as you say, using tricks like ././ makes no sense whatsoever for your use case, because you’re in control of the source code. It only makes sense when packaging a pre-built binary without chroot.
Packaging your own glibc with ld-linux.so.2 I think it’s not that awful, but will only work for a binary inside the bundle, not for the bundle runtime itself. This is similar to what you mention about including a relocated glibc inside the bundle, but I don’t know if it can be done without packaging the linker. I hope so, as packaging the linker introduces problems sometimes when invoking programs so I’d be happy to avoid it.
Have you already disregarded building static binaries? It has its own problems and it may introduce licensing issues with the LGPL, but I think it’s worth considering. I’d also like to understand how Unity packages games for Linux. They tend to be really portable, and they’re building them from a Windows IDE.
@Ismael: This has been a very interesting conversation so far :
First, regarding the bundle runtime: I firmly believe the current technique of using the libcwrap.h header file will work fine for the foreseeable future.
Regarding the actually packaged software inside the bundle, I have yet to understand what problem the libcwrap.h header is causing thiblault. He is building clutter against glibc 2.17, personally I have no problem packaging clutter in bundles using libcwrap.h, but I don’t have glibc 2.17. In other words, I still think the libcwrap.h header is a good solution for building and bundling large stacks of software, but this linker error that thiblault has experienced needs to be properly debugged.
Regarding static linking, you should know this is just technically not an option. For some smaller applications which run on bare metal, this might be an option, but it’s not an option for any larger/complex application.
This is not only an LGPL issue but really a technical one, building an application is not a matter of just building *your* source for your application, but building all of it’s dependencies into a bundle. I consider Glade to be a very *simple* application, it only requires GTK+ and libxml2, however to get there we need to build around 25 packages, many of those packages need to load modules. Pango needs to load modules to handle fonts, GdkPixbuf needs to load modules to handle image formats.
If we can get the dependency down to a low glibc dependency, handle all the needed environment variables in an AppRun script, and post process any data files installed by pango, GdkPixbuf etc to load their modules at the right path, then we have effectively reproduced a complete environment for an application to run relocatable on any system with the required glibc and the required X server protocol (even if the target host is running some ‘tiny X’ or such, some applications like Clutter applications would just require that the host’s X server have the GLX extension, easy as that).
Building glibc into the bundle itself might be too difficult (the glibc installed on a given system might make too many system dependent assumptions to be portable), so rather building an OLD glibc as a part of the bootstrapping process of creating a bundle, linking everything in that relocated build environment against an old glibc, but then NOT distributing that glibc, might be the safest approach. But still, hopefully we can get to the bottom of what happened on thiblault’s machine before abandoning the libcwrap.h header technique (which already works very well for me at the moment).
This already works well for me *as is*, building a Glade bundle is already extremely portable, all the needed modules are ready and loadable if and when some library decides to dlopen it, icons are all available, the GTK+ theme works, and building a bundle for Glade is automated in such a way that I need only issue 3 or 4 commands (one of which builds the whole stack automatically into a relocated AppDir).
The next challenge will be to bundle gedit, I think someone is already trying that. It would also be really great to have LibreOffice in the form of a bundle (even if it costs a 300MB binary download).
That is clear, libcwrap.h is a promising technique and we must push forward. The only alternatives I’ve found online are to either use a chroot, or a cross-compiler. This second option is equivalent to your backup solution involving having a secondary older glibc to link but not include in the bundle.
Regarding static linking, I was thinking more about issues linking specific dependencies like pthreads or libnss, which are known to be tricky. Building other big dependencies statically is usually not a problem. For instance linking Qt5 statically would be easy, except for the licensing issues. If modules neet to be loaded in runtime, that’s another issue.
Besides X11, I’ve also found problematic OpenGL, and in fact libGL.so.1 is the only library I don’t pack in my bundles, and expect to use the system’s one, because otherwise the bundle is granted to crash when using 3D acceleration. However I don’t think that will be a further issue; OpenGL ABI seems to be pretty stable.
Btw, you have Libre and OpenOffice bundled here: https://sourceforge.net/projects/portable/files/ I guess they’re using binary tricks, but it may be useful to compare the package you generate with the one that is generated using current techniques.
Actually linking to an older glibc seems to be pretty trivial: http://www.trevorpounds.com/blog/?p=103 (last two comments)
@Ismael: I still want to hear back from thiblault before jumping to the conclusion that we must start the build by building an old glibc and stashing it to link against every single package, but yet it’s an option.
However doing it in the way specified in that blog post is not an option (it may work by simply adjusting LD_LIBRARY_PATH to include the priority ‘old glibc’ in the linker path, though).
I should highlight the reason why I chose to use a forced include of “libcwrap.h”, instead of the possibly more “by the book” technique of specifying a custom linker script.
Plainly put, you have many packages in a stack which lead up to your application, they all need to be built using the custom non-primary glibc (or they all need to inlcude the redirected “.symver” symbols to a reasonable target version).
So, all of these packages have different build scripts, thankfully most of them use autotools, and even in those cases which do not use autotools, they respect some standard. I.e. CC and CXX can be used to override the compiler command. CFLAGS, LDFLAGS etc also can be used.
Now the tricky part: libtool.
Libtool will take your CFLAGS and LDFLAGS and make a mess of them, it will presume to be making the right educated choice of where to place your object on the linker’s command line (and we all know that the order of libraries specified at link time is very significant).
So, after experimenting with an extra .o stub and linker script, I finally settled on setting CC=’gcc -include /path/to/libcwrap.h’… why ?
Because this works *everywhere*, no matter what build scripts a given package is using, you effectively force every single source to be compiled with “libcwrap.h” as the first include (and we don’t have to maintain downstream patches on external dependency build scripts).
I completely understand we’re not closing the libcwrap.h route until we’re done fighting the fight; I’m just trying to get a complete picture of the current status quo and compare alternatives.
Thanks a lot for the enlightening explication. The more I look at it, the more it reminds me of a classic cross-compiling scenario, where systems like Yocto are often used. The same way you can use Yocto to compile a whole dependency tree leading to make your application run on the Raspberry with an armv6l toolchain, you can probably use with a x86/x86_64 toolchain with an old glibc to compile your application (or a regular x86/x86_64 toolchain but with the libcwrap.h tweak). Yocto recipes already take care of telling each particular library/application with each particular build tool (make/libtool/cmake/qmake/whatever) which cross-compiler to use. This is just an speculation and I don’t really know if it’s feasible, but the pattern matches so it may be a possible solution to study.
Anyway, recompiling the whole dependency tree with an old glibc still looks like a lot of work for the average developer (unless he’s using build services), so I really like the alternative of including your modern glibc inside the bundle. Building big libraries like Qt can take hours, and it’s not something I’d like to do unless I have very specific needs (like when working in embedded Linux)
> The symbol memcpy() changed in libc’s 2.14 ABI to behave more like memmove(),
Not quite, new glibc memcpy() on some platforms (x86_64) can do backward copy, which acts completely different from memmove().
It broke some programs, and later glibc devs added this versioning stuff. Please check: http://sourceware.org/bugzilla/show_bug.cgi?id=12518
Maybe just use memmove() in Vala instead of memcpy()?
@mh: Thanks for the clarification. I admit I did not look deeply into why the memcpy() symbol changed but found some hits on the web which suggested that it was a new function which allowed copying to overlapping memory regions (similar to memmove()).
In any case, I’m not sure where you got the idea this has anything to do with Vala.
The subject here is about compiling stacks of software in such a way that the whole stack of software requires a reasonably old version of glibc (i.e. the subject is portable application bundles).
While I’m here commenting on this post, I’d like to clarify that this technique of using the LibcWrapGenerator.vala program to generate the compatibility header is working very well for me, however @thiblault (ping ?) has run into an issue linking against clock_gettime@GLIBC_2.2.5 while building clutter on a system with a newer glibc than the one I have on my system.
@thiblault: I wanted to let the holiday season pass before starting on this again and didnt want to bother you, can you please give us an update on that issue ?
Did you solve the problem ? Is this problem due to the symbol changing location from librt.so to libc.so or such ?
As I mentioned this works very well for me (I’ve created bundles with Clutter, GTK+, Mx, libsoup… this all works fine built from my laptop) but its a high priority for me to ensure this is a viable long term solution and I’m willing to put in extra hours to make sure this works from more bleeding edge build environments.