19.09.2007 Yummy-Yummy git-svn

I’ve had to add another bit to yyhelp to make my everyday coding more convenient, so it got support for git-svn(1). The yypull and yypushpull commands will now detect a git(7) repository set up with git-svn, and unless the repository also has a regular git upstream, pulling and pushing will be carried out via git-svn rebase and git-svn dcommit. yyinfo shows the update method being used for a repository and also displays the SVN HEAD revision now if any. That, plus miscellaneous changes make version 0.6 of yyhelp:

	Overview of Changes in YummyYummySourceControl-0.6:

	* for git-svn repositories, use rebase and dcommit to pull and push
	* fixed argument parsing for yybranch and yybranchdel
	* support revision argument for yydiff
	* check revision argument to yywarp

08.09.2007 Yummy-Yummy Buglets

The YummyYummySourceControl script got a few buglet fixups during the last days, so I’ve put up version 0.5b: yyhelp.
Here’s the obligatory list of changes since the last version:

	Overview of Changes in YummyYummySourceControl-0.5b:

	* update stat info for yycommit to ignore touched but unmodified files
	* use full paths for yyrestore to fix spurious file resurrections in parent dirs
	* fixed bogus error messages on yycommit

29.08.2007 Yummy-Yummy Again

I’ve put up a new release of the slightly renamed YummyYummySourceControl script here: yyhelp (0.5).
The script still depends on cogito(7) for the implementation of yycommit, but other such dependencies have been removed at this point so it’ll eventually depend on Git(7) only. Following is a sum up of the changes between the last version and v0.5:

	Overview of Changes in YummyYummySourceControl-0.5:

	* Help improvements.
	* Support auto-push after commits.
	* Improved installation support: --install-aliases & --uninstall-aliases.
	* Automagically support less as pager.
	* Buglet fixes in yydiff.
	* Provided native implementation of yystatus with extra options.
	* New command yyinfo to list repository information.
	* New command yyblame with colorization support.
	* Allow tagging of specific revisions.

Here’s the synopsis of the new command set:

	SYNOPSIS
	    yyadd          [FILES…] - add files to git repository
	    yyblame        [FILES…] - annotate file source code
	    yybranch    [-f] <branch> - add branch (forcable)
	    yybranchdel [-f] <branch> - delete branch (forcable)
	    yyChangeLog               - show git log in ChangeLog style
	    yycommit       [FILES…] - commit current working tree
	    yydiff         [FILES…] - show committable differences in
	                                working tree
	    yygc                      - repack and prune repository
	    yyhelp                    - display yy* help information
	    yyinfo                    - display repository information
	    yylsbranches              - list branches
	    yylstags                  - list tags
	    yypull                    - pull upstream sources
	    yypushpull                - push & pull upstream sources
	    yyremove       [FILES…] - remove files from git repository
	    yyreset                   - reset (revert to HEAD) all files in the tree
	    yyrestore      [FILES…] - forcefully re-checkout specific files
	    yystatus       [-t|-u|-c] - display working tree status
	                                -t: trim unknown; -u: show unknown;
	                                -c: trim and use commit message style
	    yytag    <tag> [revision] - add tag
	    yytagdel <tag>            - delete tag
	    yyuncommit                - undo the last commit (must be unpushed)
	    yyview                    - browse & navigate the history
	    yywarp           <branch> - checkout new branch

17.08.2007 What are ELF libraries good for anyway?

Or: Who in the world is actually linking against GLib without Gtk+?

I’m currently in the process of building some general purpose infrastructure for Rapicorn and because a good portion of the C++ utilities used in Beast and Rapicorn are factored out into an extra library called Birnet. I’m running into similar complexity issues here that we encountered with libglib vs. libgobject vs. libgdk already. I.e. lower level modules cannot reference or use mechanisms provided at a higher level (for instance deriving GHashTable from GObject). This becomes even worse with C++ which lacks support for partial classes (a class definition spanning multiple source files, like implementing gtk_widget_foo() outside of libgtk).

One remedy would be to actually merge libbirnet and librapicorn into a single project and let Beast depend on librapicorn. However, that’s essentially like we started out with GLib and Gtk+ in late 1996 when it got split out of the Gimp package. In June 1998 we split off libglib into a separate package for programs needing C utilities but not libgtk. So now, I’m reconsidering this separation after almost a decade. Reflecting on the move is probably not a bad idea before attempting the opposite with Rapicorn:

How many programs are actually linking against libglib but not libgdk, and is there any real benefit from separation?
If your machine has libglib and a few spare minutes, you can use this command line to find out on your own:

	find `echo $PATH | sed 's/:/ /g'` -type f -perm +111 | xargs ldd 2>/dev/null |
	  awk '/\<libglib-2\>/{ lg+=1 } /\<libgobject\>/{ lo+=1 } /\<libgdk_pixbuf\>/{ lp+=1 }
               END { print "glib=" lg " gobject=" lo " gdk=" lp ;
                     print "glib-only:    " 100*(lg-lo)/lg "%";
                     print "gobject-only: " 100*(lo-lp)/lg "%";
                     print "gdk-lot:      " 100*lp/lg "%"; }'

Here are results from a couple machines I had quick access to:

	Site                       Gdk  GObject     GLib  #Apps
	gimp.org:                50.0%    12.5%    37.5%     16
	gtk.org:                 47.1%    17.6%    35.3%     17
	developer.gnome.org:     19.5%    53.7%    26.8%     41
	my server (sarge):       66.7%    14.1%    19.3%    192
	my laptop (etch):        72.7%    14.8%    12.4%    209
	my desktop (feisty):     72.2%    17.1%    10.7%    252
	64bit KDE desktop (sid): 53.0%    16.7%    30.2%    338

That is, the servers have a quite limited set of GUI applications installed, but across a full fledged desktop the vast majority of applications is linked against libgdk anyway. I found Stefan Westerfelds Amd64 KDE desktop particularly interesting: It actually has the most applications linking against GLib, prolly because it has Mono and libQtCore.so installed which both link against GLib these days. Does anyone actually have a system with libglib installed but not libgdk?

Next, let’s take a look at the actual “savings”:

	ls -l libglib-2.0.so.0.1200.4 libgmodule-2.0.so.0.1200.4 \
              libgobject-2.0.so.0.1200.4 libgthread-2.0.so.0.1200.4
	-rw-r--r-- 1 root root 596608 2006-11-16 10:26 libglib-2.0.so.0.1200.4
	-rw-r--r-- 1 root root   9784 2006-11-16 10:26 libgmodule-2.0.so.0.1200.4
	-rw-r--r-- 1 root root 237156 2006-11-16 10:26 libgobject-2.0.so.0.1200.4
	-rw-r--r-- 1 root root  14028 2006-11-16 10:26 libgthread-2.0.so.0.1200.4
	=====================================
	                       857576 = 837KB

	ls -l libatk-1.0.so.0.1214.0 libcairo.so.2.9.2 libpango-1.0.so.0.1400.8 \
	      libgtk-x11-2.0.so.0.800.20 libgdk_pixbuf-2.0.so.0.800.20 \
              libgdk-x11-2.0.so.0.800.20
	-rw-r--r-- 1 root root  102504 2007-03-14 13:44 libatk-1.0.so.0.1214.0
	-rw-r--r-- 1 root root  395836 2006-10-20 07:44 libcairo.so.2.9.2
	-rw-r--r-- 1 root root  235464 2007-01-14 22:27 libpango-1.0.so.0.1400.8
	-rw-r--r-- 1 root root   88604 2007-03-04 22:21 libgdk_pixbuf-2.0.so.0.800.20
	-rw-r--r-- 1 root root  528580 2007-03-04 22:21 libgdk-x11-2.0.so.0.800.20
	-rw-r--r-- 1 root root 3043524 2007-03-04 22:21 libgtk-x11-2.0.so.0.800.20
	=======================================
	                       4394512 = 4292KB

GtkGdkPangoAtkCairoGlib / GLib ratio: (4394512 + 857576) / 857576 = 6.12.
So the “savings” turn out to be splitting off 5/6th of the GUI stack size. At best, a GLib-only program is saving 4.2MB that way. But then again, those are likely to be in memory already anyway. And most definitely, they reside in a library, available as const .text on the harddisk. To put that into perspective: We’re talking about only a handful megabytes here. In physical size actually less than a hires desktop background image, smaller than some icon themes, or roughly 10% of a full linux-2.6.8 kernel + modules binary footprint (41MB). Also symbol resolution performance doesn’t necessarily improve through splits, Ulrich Drepper has a few words on multiple DSOs during deployment.

Given that selecting Gtk+ subcomponents to allow selective shrinkage for size constrained embedded devices via configure.in options is on our TODO list anyway; for this set of libraries: Size is not worth a split!

Of course, other aspects not reflected in library size weigh in much more strongly. Such as the ability to build different user communities around projects (e.g. for Cairo vs. Glib) or the reduced dependency chain (sshfs and Qt wouldn’t depend on GLib if it drew in Pango or Gtk+).
So while for the vast majority of applications the GLib split off doesn’t make a huge difference technically, software evolution arguments make all the difference, definitely justifying a separation. The GLib internal splits are definitely not worth the hassle and conceptual boundaries though, if we ever break binary compatibility again, merging the GLib internal libraries and respective pkg-config packages would be a good idea.

Back to my original problem… I’m now pretty convinced that merging Rapicorn with the Birnet utility library will most likely not pose a problem anytime soon. And Qt 4 kindly demonstrates that splits can also be carried out successfully later on during the game.

17.07.2007 OpenGL for Gdk/Gtk+

The idea of using OpenGL as a future core rendering architecture for Gtk+ has been brought up a couple times at GUADEC (and then some variations thereof). However there are good reasons to avoid that and major issues with the suggested approaches, in particular these need to be considered:

1) A dependency on OpenGL for a library as portable and as widely used as Gtk+ these days, could only ever be a weak one. That is, OpenGL features may or may not be used, depending on whether the current platform a Gtk+ application runs on actually has OpenGL available or not (e.g. by animating widgets conditionally to only be carried out when acceleration support is present).

2) The OpenGL 2D drawing API is effectively unusable for Gdk/Gtk+ drawing primitives. The main problem here is that OpenGL doesn’t provide pixel-perfect 2D drawing operations which are necessary for accurate input event processing and a coherent visual presentation (it also doesn’t always provide anti-aliasing in the 2D drawing API either). Here is a very good web page with nice screenshots summarizing the problems with OpenGL pixel-perfectness: OpenGL: “not pixel exact”, Hardware AntiAliasing.

3) By using XRENDER and hardware-accelerated X drivers, Cairo is already being performance optimized to utilize hardware acceleration. Trying to use a portable OpenGL subset instead (pixel shaders / triangle rendering) would be fairly pointless, it’d effectively be using the available portable hardware acceleration facilities through another indirection. So with more and more Gtk+-based platforms/applications moving to Cairo-based drawing, there is no additional infrastructure or support code needed to make use of available hardware acceleration facilities. Essentially, the portably usable hardware acceleration subset is brought to you automatically through Cairo and X.

For the lazy, here’s a quick overview of the artifacts presented on the OpenGL comparison page:

.

13.07.2007 Switch On Strings In C And C++

For Rapicorn i need a simple and nicely readable way to special case code on various strings, which isn’t a problem in most scripting languages but is not provided by C/C++. Switching on strings turns out to be a widely investigated topic: SOS (agay), TheScript, JavaBug (CNFW), CodeGuru, CLC-FAQ and google has lots more.

Now here goes my approach to switching on strings for C/C++, caveats upfront:
- The strings may only consist of identifier chars: [A-Za-z_0-9];
- Convenient application is based on a GNU Preprocessor feature: -imacros;
- Macro implementations use a GCC-ism: Statement Expressions;
- The convenient command line variant uses a bash-ism: Process Substitution;
- One compile-time script is needed, currently implemented in python (though it could be written in any language).

With this out of the way, let’s dive into the code. This is the syntax:

	switch (SOSID_LOOKUP (sample_string))
          {
          case SOSID (hello): printf ("Hello ");   break;
          case SOSID (world): printf ("World! ");  break;
          case 0: default:    printf ("unknown "); break;
          }

This is how the code needs to be compiled: gcc -Wall -O2 -imacros <(sosid.py <input.c) input.c

The actual implementation is fairly simple, sosid.py extracts all identifiers from SOSID() statements and generates a handful of macro definitions: SOSID(), SOSID_LOOKUP(), SOSID_LOOKUP_ICASE(), ... Plus input specific macros: SOSID__hello, SOSID__world, ... GCC will pick up those definitions via the -imacros option. That is enough to implement SOSID_LOOKUP() so that it can yield the integer IDs that the SOSID(*)/SOSID__* statements expand to for any string that corresponds to one of the SOSID(identifiers) occurring in the source file.

Example:
call_switch ("hello"); call_switch ("foo0815"); call_switch ("world");
Yields:
Hello unkown World!

The lookups are implemented via a binary search and the strings are sorted in a way so that binary searches for case sensitive and case insensitive lookups are both possible. That means for large string lists, the lookups which are of complexity O(ld N) actually have a performance advantage over lengthy if..else if..else constructs with O(N) complexity.
The code is available as a single script at the moment: sosid.py.

Note that the bash-ism, imacros-use and GCC-isms are not mandatory, ordinary temporary files can be generated by the script and be included instead and it could be adapted to generate portable inline functions. The identifier-chars restriction is a hard one though. It comes from the fact that SOSID(ident) must yield valid C/C++ integers, and that can only be accomplished by token pasting. Depending on GCC is good enough for Rapicorn, so people will have to express active interest in this, in order for me to make the script more portable.

29.06.2007 YummiYummiGit

Starting with the discontinuation of cogito, i began to wonder about my future git usage patterns. On the one hand, i have become quite used to the convenience of some of the cogito commands, on the other hand i found myself using or looking up git commands every other day. Since the discontinuation, cogito cannot be expected to remedy those work-flow interruptions any time soon, so i actually started to cook up my own small set of convenience wrappers. I’m adding those on demand whenever i need a new source repository command, and i try hard to keep it simple, avoid giving the wrappers any options and keep an eye on them being very shell-completion friendly.

The GPLv3 licensed result is a single shell script plus a handful links for invocation: yyhelp.

For the curious, the prefix ‘yy’ was chosen to allow conflict free shell completion. I’ll quote the yyhelp information here, so people can decide if it’s interesting for them and also get to see the installation instructions. I’m not entirely sure where to take this shallow wrapper in the future, however keeping it simple is really the main incentive behind it. Any deeper logic required should rather be filed as git-core requests. If you want to talk to me about YummiYummiGit, drop me a line via email, or try #git on irc.freenode.net:6667.

	YummiYummiGit                                                     YummiYummiGit

	NAME
		YummiYummiGit - The simplest git wrapper ever, yet convenient

	SYNOPSIS
		yyadd          [FILES…] - add files to git repository
		yybranch    [-f] <branch> - add branch (forcable)
		yybranchdel [-f] <branch> - delete branch (forcable)
		yyChangeLog               - show git log in ChangeLog style
		yycommit       [FILES…] - commit current working tree
		yydiff         [FILES…] - show committable differences in working tree
		yygc                      - repack and prune repository
		yyhelp                    - display yy* help information
		yylsbranches              - list branches
		yylstags                  - list tags
		yypull                    - pull upstream sources
		yypushpull                - push & pull upstream sources
		yyremove       [FILES…] - remove files from git repository
		yyreset                   - reset (revert to HEAD) all files in the tree
		yyrestore      [FILES…] - forcefully recheckout specific files
		yystatus                  - display working tree status
		yytag               <tag> - add tag
		yytagdel            <tag> - delete tag
		yyuncommit                - undo the last commit (must be unpushed)
		yyview                    - browse & navigate the history
		yywarp           <branch> - checkout new branch

	HISTORY
		YummiYummiGit was created as a very shallow wrapper around git-* tool
		option variants, to simplify common cases.
		Depending on programming habits, YummiYummiGit may or may not suit your
		daily needs. It is in any case not meant as a full replacement for the
		git interface (there is e.g. no yyclone) and subject to change.

	INSTALLATION
		To install YummiYummiGit, copy yyhelp into a bin/ directory from $PATH
		and invoke: ./yyhelp –create-aliases

	YummiYummiGit-0.4                                                 YummiYummiGit

28.06.2007 Searchable API Docs

Newly built GLib & Gtk+ API docs are online now. In the process, i fixed a long standing documentation feature request: searchable API docs. This turned out to be the most requested documentation feature at the GNOME booth LinuxTag 2007.

The documentation updates cover Gtk+-2.11.4, so they contain new features like: GtkBuildable, GtkBuilder, GtkScaleButton, GtkVolumeButton and GtkTooltip.

13.06.2007 Back Online

I just got back online after a full week of DSL-outage due to the Deutsche Telekom Strike. If you are awaiting response via email or bugzilla from me, please be patient, I’m doing my best to catch up. Feel free to poke me about urgent issues via IRC/IM though, or drop me a reminder if you haven’t heard back from me in another week.

04.06.2007 LinuxTag

As announced by Hallski, Sven, Mitch and me went to LinuxTag 2007 and operated the Imendio and GNOME booths. As usual, Sven did a great job nurturing random users approaching the booth, together with Mathias Hasselmann and Michael Köchling. I put more focus on the amazingly large program of the conference, of which I’ll give a short roundup here.

The Xen keynote was on Wednesday. Xen 3.0 comes with some interesting new features, it’ll introduce IO virtualization and supports new virtuallization technologies from Intel and AMD. One important lesson I took away from that session is that using virtualization aware drivers on the guest OS can boost performance from roughly 10% using generic virtualization techniques, to more than 90% of the ideal performance throughput (native host OS performance).
-
This day, there was also the ZODB3 talk. This is an object database which can be used completely independently from Zope, it provides a very nice interface in python to implement hierarchical object tree persistence, has ACID transactions with rollbacks and allows for doomed transactions. At the lower level it uses a stable subset of python’s pickle and supports multiple storage backends.

Thursday had a Linux forum where Thomas Gleixner discussed recent realtime work in the kernel. The low latency and preemption patches that went into the kernel over the 2.6.x series brought a number of positive side effects such as general responsiveness improvements on loaded systems and new debugging mechanisms. From the new debugging facilities and raised threading issue awareness amongst kernel developers, a good 1200 patches containing bug fixes and cleanups resulted. And development in this are has not come to a halt, current/future work includes getting rid of idle-state interrupts that do nothing by having a tick-less kernel that only wakes up every once in a while when actual work is due and cleaning up general jiffie dependent code in the kernel. Now what’s left to hope is that distributors get their act together and enable the low-latency preemption patches for their desktop kernels. The patches work, they are stable, and they provide a much better user experience. E.g. if you experience sluggish system behavior during crypto filesystem IO, or experience drop outs with your sound system/server, don’t blame it on the kernel people, blame it on your distribution not allowing time slice preemption.
-
The other talk I attended was held by Matthias Hopf about the Compiz+Beryl merger. The resulting effects he presented excited the crowd as usual, and then he talked some about ongoing developments like input event transformation. After the talk we had some more personal chatter about using the 32bit XRGB visual to add alpha channels to XWindows, and future X extensions to allow applications to notify the server about when to issue EXT_texture_from_pixmap (needed for flicker free composite support).

On Friday, Lennart gave a very good presentation on the state of PulseAudio. He described how it solves the vast majority of audio use cases and can in combination with libsydney finally put an end to the never-ending lack of a portable and usable audio API. Beyond the talk, Lennart and I used every spare minute during the LinuxTag days to discuss libsydneys new API. All in all, it looks like a suitable candidate to replace (or continue) the effort Stefan and I started with CSL (we later suspended the project assuming PortAudio would fulfil the role) to make sound backends transparent.
-
After lunch, Quim Gil presented present and future of the Maemo platform. The points I personally found most notable are:
* libhildon2 is going to become an upstream community project, using Gnome infrastructure like bugzilla, with Nokia providing the core developers.
* Future platform updates (applications and OS) should be possible via APT, so flashing becomes a secondary upgrade method.
* Nokia is currently collecting feature requests for the Maemo platform. They’ll be integrated into Nokias platform plans where possible, so if you have any input to provide, state it here: Maemo Roadmap Page.
* The Maemo versioning scheme now uses alphabetical letters to indicate versioning progress. The current/upcoming versions are:
- [B]ora: current platform (3.x);
- [C]hinook: next platform (4.x) based on Gtk+-2.10, comes with SDKs before launch;
- [D]iabolo: intended to keep API/ABI from here on, unless upstream also breaks;
- [E]lephanta: SBox2 might be available at this time.
* Nokia tries to open up as much of the Maemo platform as possible and they will try to reduce dependencies between opened software and closed platform components in the future. The reason that some programs stay closed anyway are: a) missing legal clearance or licensing for some code portions - opening up code in one division of the company might affect legal claims (IP) on closed components developed and supported in another division); b) code may be subject to hardware vendor NDAs.
* Nokia will embrace attempts such as getting Maemo to run on different hardware in the future, or running non-Maemo software on the N770/N800.
* Nokia intends to introduce abstraction layers for hardware specifics (N770 vs. N800 vs. future devices) where possible.
-
The closing talk for this day was the Gimp presentation by Simon Budig, titled “Pimp My Gimp”. Simon presenting new Gimp features has become somewhat of a LinuxTag tradition over the years, and as usual new Gimp features managed to excite the crowd, even though we also had the usual glitches like a perfectly tested new clone-tool refusing to work on stage. ;-)

Saturday morning started with a vivid presentation on DBus by Marcel Holtmann. Though no stunning new features were presented, he did a good job on introducing the overall architecture and getting the audience hooked up for DBus applications with his presentation and some simple example code.
-
After noon, there was a big podium discussion on preventive data mining of personalized records, currently planned to be realized in upcoming laws by the German government. While the forum was quite interesting and definitely necessary to have at a conference like this, the discussion didn’t present surprisingly new findings for anyone following matters already. The discussion was great however, mostly because of the strong involvement of a rather large audience. It was pointed out that there is a massive lack of public awareness for the incredible data mining hunger of the government organizations and certain big companies, and that this is one of the topics that are rather uneasy to educate the majority of the democratic republic on.
-
Later during the day, Jono Bacon gave his obligatory Ubuntu talk about how Ubuntu cares about user experience and about growing the community. Personally, I still feel that more involvement from Canonical in genuine upstream development such as e.g. the Gtk+ project would be better in the long run for both Canonical and the upstream eco systems.
-
Finally, there was the Open Source Press talk which started out on how CopyLeft used to be the natural state for information exchange since the beginning of mankind, until publishers came into existence in the 16th century and invented CopyRight restrictions to preserve their investments in terms of hardware and human resources. It then was suggested that in a completely networked future computer age, we’ll be back to a pure CopyLeft situation. The presenter didn’t really seem to apply his findings though, when in the second half of the session he assumed that all mass-digestible texts must be edited by publishers and thus unconditionally require copyright laws. He also didn’t want to acknowledge that books existed which are sold and published online or that large amounts of interesting text (>= 500 pages) could possibly be written by non-publishers. While this presentation showed that publishers do recognize and ponder about open content these days, it also made clear that they still have a lot to learn.

As a closing note, I’d like to say a big Thank You to the LinuxTag Orga-Team. All throughout the four days there was an exceptionally good program in five presentation rooms, with up to four additional forum presentations distributed across the booth areas. From our perspective overall event organization went pretty well, this also resonated in German media.

Bad Behavior has blocked 130 access attempts in the last 7 days.