Category: General

I’ve been playing with an idea of how to extend the traditional unix shell pipeline. Historically shell pipelines worked by passing free-form text around. This is very flexible and easy to work with. For instance, its easy to debug or incrementally construct such a pipeline as each individual step is easily readable.

However, the pure textual format is problematic in many cases as you need to interpret the data to work on it. Even something as basic as numerical sorting on a column gets quite complicated.

There has been a few projects trying to generalize the shell pipeline to solve these issues by streaming Objects in the pipeline. For instance the HotWire shell and Microsoft PowerShell. Although these are cool projects I think they step too far away from the traditional interactive shell pipelines, getting closer to “real” programming, with more strict interfaces (rather than freeform) and not being very compatible with existing unix shell tools.

My approach is a kind of middle ground between free-form text and objects. Instead of passing free-form text in the pipeline it uses typed data, in the form of glib GVariants. GVariant is a size-efficient binary data format with a powerful recursive type system and a textual form that is pretty nice. Additionally the type system it is a superset of DBus which is pretty nice as it makes it easier to integrate DBus calls with the shell.

Additionally I created a format negotiation system for pipes such that for “normal” pipes or other types of output we output textual data, one variant per line. But, if the destination process specifies that it supports it we pass the data in raw binary form.

Then I wrote some standard tools to work on this format, so you can sort, filter, limit, and display variant streams. For example, to get some sample data I wrote a “dps” tool similar to ps that gives typed output.

Running it prints something like:

$ dps
 <{'pid': <uint32 1>, 'ppid': <uint32 0>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'systemd'>, 'cmdline': <'/usr/lib/systemd/systemd'>, 'cmdvec': <['/usr/lib/systemd/systemd']>, 'state': <'S'>, 'utime': <uint64 38>, 'stime': <uint64 138>, 'cutime': <uint64 3867>, 'cstime': <uint64 1273>, 'time': <uint64 1344635046>, 'start': <uint64 1>, 'vsize': <uint64 61488>, 'rss': <uint64 24408>}>
 <{'pid': <uint32 2>, 'ppid': <uint32 0>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'kthreadd'>, 'cmdline': <'[kthreadd]'>, 'state': <'S'>, 'utime': <uint64 0>, 'stime': <uint64 1>, 'cutime': <uint64 0>, 'cstime': <uint64 0>, 'time': <uint64 1344635046>, 'start': <uint64 1>, 'vsize': <uint64 0>, 'rss': <uint64 0>}>
 ...

Not super-readable, but its a textual format that you could combine with traditional tools like grep and awk.

But, with the type information we can do more interesting things. For instance, we could filter using a numeric comparison, say finding
all system uids:

$ dps | dfilter euid \< 1000
 <{'pid': <uint32 1>, 'ppid': <uint32 0>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'systemd'>, 'cmdline': <'/usr/lib/systemd/systemd'>, 'cmdvec': <['/usr/lib/systemd/systemd']>, 'state': <'S'>, 'utime': <uint64 38>, 'stime': <uint64 139>, 'cutime': <uint64 4290>, 'cstime': <uint64 1318>, 'time': <uint64 1344635266>, 'start': <uint64 1>, 'vsize': <uint64 61488>, 'rss': <uint64 24408>}>
 <{'pid': <uint32 2>, 'ppid': <uint32 0>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'kthreadd'>, 'cmdline': <'[kthreadd]'>, 'state': <'S'>, 'utime': <uint64 0>, 'stime': <uint64 1>, 'cutime': <uint64 0>, 'cstime': <uint64 0>, 'time': <uint64 1344635266>, 'start': <uint64 1>, 'vsize': <uint64 0>, 'rss': <uint64 0>}>
 ...

Then we can add numerical sorting:

$ dps | dfilter euid \< 1000 | dsort rss
 <{'pid': <uint32 1>, 'ppid': <uint32 0>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'systemd'>, 'cmdline': <'/usr/lib/systemd/systemd'>, 'cmdvec': <['/usr/lib/systemd/systemd']>, 'state': <'S'>, 'utime': <uint64 38>, 'stime': <uint64 139>, 'cutime': <uint64 4290>, 'cstime': <uint64 1318>, 'time': <uint64 1344635365>, 'start': <uint64 1>, 'vsize': <uint64 61488>, 'rss': <uint64 24408>}>
 <{'pid': <uint32 769>, 'ppid': <uint32 745>, 'euid': <uint32 0>, 'egid': <uint32 0>, 'user': <'root'>, 'cmd': <'Xorg'>, 'cmdline': <'/usr/bin/Xorg :0 -background none -logverbose 7 -seat seat0 -nolisten tcp vt1'>, 'cmdvec': <['/usr/bin/Xorg', ':0', '-background', 'none', '-logverbose', '7', '-seat', 'seat0', '-nolisten', 'tcp', 'vt1']>, 'state': <'S'>, 'utime': <uint64 1602>, 'stime': <uint64 3145>, 'cutime': <uint64 22>, 'cstime': <uint64 9>, 'time': <uint64 1344634285>, 'start': <uint64 1081>, 'vsize': <uint64 108000>, 'rss': <uint64 16028>}>
 ...

And, nice display:

$ dps | dfilter euid \< 1000 | dsort rss | dhead 4 | dtable pid user rss vsize cmdline
 pid     user      rss    vsize  cmdline
   1   'root'    24408    61488 '/usr/lib/systemd/systemd'
 769   'root'    16028   108000 '/usr/bin/Xorg :0 -background none -logverbose 7 -seat seat0 -nolisten tcp vt1'
 608   'root'    15076   255312 '/usr/bin/python /usr/sbin/firewalld --nofork'
 747   'root'     8276   452604 '/usr/sbin/libvirtd'

Note how we do two type-sensitive operations (filter by numerical comparison and numerical sort) without problems, and that we can do the “head” operation to limit output length without affecting the table header. We also filter on a column (euid) which is not displayed. And, all the data that flows in the pipeline is in binary form (since all targets support that), so we don’t waste a time re-parsing it.

Additionally I think this is a pretty nice example of the Unix idea of “do one thing well”. Rather than having the “ps” app have lots of ways to specify how the output should be sorted/limited/displayed we have separate reusable apps for those parts. Of course, its a lot more typing than “ps aux”, which makes it less practical in real life.

I’ve got some code, which while quite rudimentary, does show that this could work. However, it needs a lot of fleshing out to be actually useful. I’m interested in what people think about this. Does it seem useful?

Last week I landed a new feature in Glib that I’ve been wanting to do for a long time: A resource framework. Resources are things that are naturally part of an application or library, but not really normal code. For instance, our code increasingly uses xml  to describe user interfaces and menus.

Traditionally these either had to be manually inserted into the code, like so:

static const gchar *ui_info =
"<ui>"
"  <menubar name='MenuBar'>"
"    <menu action='FileMenu'>"
"      <menuitem action='Quit'/>"
"    </menu>"
"  </menubar>"
"</ui>";

Or a file was stored in /usr/share/$application/ and you had to write code to manually find and load the file, and cache it if used often. This is not a lot of code, but it can be tricky as all I/O code needs to handle errors and the external file makes it harder to make the library/app relocatable.

Instead, with resources you store your data file as plain files in your source tree, edit them with your favourite editor, with full syntax highlighting, automatic indentation, etc. Then you reference these files from a resource description file (another xml file) and use glib-compile-resources to compile these into a binary chunk which is linked into the application.

The resource framework then automatically registers all such resource bundles in a global namespace, where you can quickly look up resource data by a resource path. There are API calls to get the data as a direct pointer, as well as streaming data, but in most cases there are helper functions that let you just specify the pathname. So, for instance, you can just do:

 gtk_builder_add_from_resource (builder, "/org/gnome/appname/menu.ui", &error);

Which would handle all the work for you. And while this looks like an I/O operation its really just a hashtable lookup on linked-in data, reading from the (shared, readonly) data section in your executable, so its very fast and safe.

Additionally there are some tricks the resource compiler can do for you. For instance, you can specify that a resource should be compressed, which means that the data is stored compressed, and the APIs uncompress for you automatically. You can also specify that xml files should be pre-processed to strip away whitespace, which avoids wasting time and memory on something that is not useful at runtime.

There is also support for resource:// URIs, which means you can easily reference resource data like icons from e.g. CSS and UI files.

On Linux we use some gcc specific extensions to store the resources in separate ELF sections, which means its very easy to extract resource data from the binaries. Glib even ships with a tool that lets you do this:

$ gresource list libgtk-3.so
/org/gtk/libgtk/cursor/dnd-ask.png
/org/gtk/libgtk/cursor/dnd-copy.png
/org/gtk/libgtk/cursor/dnd-link.png
/org/gtk/libgtk/cursor/dnd-move.png
/org/gtk/libgtk/cursor/dnd-none.png
/org/gtk/libgtk/gtk-default.css
/org/gtk/libgtk/gtk-win32.css
$ gresource extract libgtk-3.so /org/gtk/libgtk/gtk-default.css
@define-color fg_color #000;
@define-color bg_color #dcdad5;
@define-color text_color #000;
...

If you’re interested in using resources in your application, check out the documentation, or look at this example commit that converts Nautilus to use resources.

The last few weeks I have been working on the Gtk+ win32 port. Since the client side windows work landed in Gtk+ 2.18 the Windows port has been a bit broken, but now I finally sat down and fixed the remaining issues. So, the newly released Gtk+ 2.24.8 is now officially the best ever Gtk+ 2.x release on windows.

Then I forward ported all the work to the current Gtk 3 tree. It was mostly trivial, but one thing that changed a lot in Gtk3 is theming. The old windows theme was mostly a custom themeing engine, but in Gtk+ 3 we want to use engines less in favour of CSS. So, I had to completely redo the windows theme using CSS.

I added a few CSS extensions that access the win32 theming APIs, so you can get theme backgrounds, theme colors and theme sizes. Then the rest is just traditional CSS to bind the things together.

Here is an image of the current state:

There are clearly still some issues that need fixing, but it works impressively well for just being some CSS. Check it out.

After a vacation and an excellent week in Berlin at the Desktop Summit I’m now back working on Gnome Contacts.

I’m slowly getting most of the UI mockups implemented in preparation for the UI freeze next week, and the evolution-data-server backend in libfolks has landed and been made the default. We also have support in evolution-data-server for the new Online Accounts system. This means we’re currently in a pretty good shape for Gnome 3.2, although there is still a lot of work to do, like getting contacts linking working.

However, today I’d like to talk about something else that I’ve been working on. As you can see in my last update Gnome Contacts relies a great deal on avatars in the UI, and its often the case that your address book contains a lot of people that don’t have one. All these default avatar icons makes it hard to scan the contact lists.

To solve this I’ve been talking to the guy behind the awesome robohash site. It turns out that he’s actually somewhat of a Gnome fan, even helping out with news.gnome.org a long time ago. He’s agreed to let us use the source images and code (in exchange for pimping robohash.org, which you should use! It’s awesome!). I’ve been working on turning the code and images into a library that can be used locally by gnome-contacts. It’s still a work in progress, but here is a screenshot:

This is still a work in progress, but I’m very hopeful that we can land this for Gnome 3.2.

With Gnome 3.0 being released I had some time to spend on the broadway Gtk+ backend.

I managed to get rid of all roundtrips, which should make remote access snappier. I also fixed a bunch of bugs and added some missing features (including in-process cut and paste).

All that stuff  is nice, but not really all that interesting to show off.

However, I also implemented a cool idea I’ve had for a while about using chromeless browser windows with a canvas inside to get real toplevel windows. It turns out that this works pretty well. Although you have to disable the dom.disable_window_feature_open.location config option to get rid of the location bar at the top of each window.

Here is a short video showing how this looks: (original webm source here)

[vimeo width=”512″ height=”384″]http://vimeo.com/22092069[/vimeo]

Cool, eh? Its all in git, go play with it.

PS: I have no idea why google thinks I’m in france…

I can’t believe we finally released 3.0! The last months have been crazy with energy. The whole project feels revitalized.

Also, I’ve been using gnome-shell now for some time, and I’m really liking it. Going back to gnome 2 now feels clumsy and painful. You should all try it out and make up your own mind, but please try to avoid immediately tweaking it to your old ways, instead try the new defaults for a while, you might like it more than you expect.

Make sure to read the cheat sheet to learn all the new tricks. I really like the new alt-<key over tab> feature!

I recently switched to using gnome-shell in my day to day work, and so far I like it a lot. However, I use a dual monitor setup, and the gnome-shell support for multihead was very rudimentary.

Here is an example desktop with a 1920×1080 primary monitor and a 1280×1024 monitor on the left:

And the corresponding overview:

There are a number of issues here:

• All previews are on a single monitor. So, while we have a lot of space for normal windows with two monitors we need to cram them all into part of a single monitor in overview mode. This makes the windows way to small to be usable.
• Due to internal issues the part of the overview that shows the windows is always the same aspect ratio as the screen (i.e. the total size of all monitors). This is fine in the case of one monitor, but with two monitors it doesn’t match the monitor aspect ratio making windows even smaller.
• During the overview switch the windows on the extra monitor fly a long way, making the animation not work really well, as well as breaking the natural grouping of related windows being on the same monitor.
• The extra monitor space is unused
• The thumbnails that normally slide in from the side in a slick way instead overlap the other monitor and its hard to make slide in since the mouse enters the other monitor instead.
• The thumbnails include the “dead” area due to the different sized monitors and generally look weird.

Additionally, but not visible here:

• Its hard to hit the hot corner on the primary monitor if there is a monitor to the left. (And to hit the menu in the right corner if there is a monitor to the right.)
• If the top-left monitor is not the primary there is no hot corner in the easy to reach corner, instead you have to mouse into the primary and then try to hit the corner.
• If another monitor is taller than the primary, its very hard to hit the message tray on the bottom of the primary monitor, as the pointer passed over to the dead area.
• In most typical cases the external monitor is used to show something static, like always visible information, a presentation on a projector, or a movie on a TV. However, the extra monitors are connected to the same virtual workspaces as the primary, so when you switch workspace the apps on the extra monitors switch around too.

The last few weeks I’ve been working on this, and these changes have now landed in git.

Here is how it looks now:

And the overview:

We can see here:

• There is a window overview on each monitor, showing the windows on that monitor.
• The window overview area is not artificially constrained by aspect ratios but as large as it can be, so the windows are larger in the overview.
• The thumbnails are always slided in if there is a monitor to the right of the primary monitor.
• The thumbnails only show the primary monitor. This is because the whole workspace feature only affects the primary monitor. Windows on extra monitors stay in place when switching workspaces, only the primary monitor changes.

Additionally some things we can’t see:

• Every monitor with a natural top-left corner (i.e. the top left monitor, or a monitor to the right that is taller but aligned at the bottom) gets a hot corner that activates the overview.
• The ever-rocking ajax has added pointer barrier support to the Xserver for us. If this is available (he backported it to Fedora 15, yay!) we add one-sided barriers in the panel and the message tray so that its easier to hit these, even if there are monitors on the sides.
• Additionally, as part of the pointer barrier support the Xserver now automatically blocks the mouse from entering the dead areas that are not shown on any monitor. This is great if monitors have different resolutions.

These are not enormous changes, but the difference in day to day use in a multimonitor setup is like day and night. These should be in the next release which is out soon, so if you’re a multimonitor user, do try it out!