next hack: timeouttop

i had a ‘future work’ item in my last post. i couldn’t resist doing it so now it’s done.

timeouttop is a program that when coupled with the kernel patch from my last entry will show a top-like display of which processes are being most frequently woken by timeouts.

notes: it works by keeping track of the state of the counters from up to 60 seconds ago and comparing it with the current state. if a process isn’t in both of the lists then it is not shown. this means that if a new process (just started) wasn’t running 60 seconds ago then it won’t show in the list until it has run for the full 60. this could be considered a bug.

the progrram also assumes a terminal width of 80 and a length of however long it needs to show all of the processes with non-zero timeouts in the sample period. the terminal is assumed to support vt100-like codes for clearing the screen.

good enough for now :)

today’s hack: pstimeouts

today i wrote a kernel patch to find out just how often processes are waking up as a result of setting timers for themselves. the intention is to provide a tool to make it extremely easy to spot poorly behaved applications.

the kernel patch recognises 5 types of “timers” that might cause a process to wake up:

  • poll returning due to timeout after sleeping
  • select, same as above
  • epoll, same as above
  • the ‘real time’ interval timer (SIGALRM)
  • anything else inside the kernel that uses schedule_timeout

the kernel patch makes this information available in /proc/pid/timeouts as 5 numbers (corresponding to the list above).

i wrote ‘pstimeouts’ as a small utility to read this data and present it to the user. it’s written in straight-up C with no library dependencies (not even glib) as to be usable by the biggest number of people.

here is a ‘screenshot’:

desrt@acquiesce:~$ pstimeouts
  pid timeouts process
 4568       44 x-session-manager
 4607        7 /usr/bin/dbus-daemon
 4609      704 /usr/lib/libgconf2-4/gconfd-2
 4612        0 /usr/bin/gnome-keyring-daemon
 4615     6615 /usr/lib/control-center/gnome-settings-daemon
 4629     5428 /usr/bin/metacity
 4634    39574 gnome-panel
 4636    17635 nautilus
 4639     1391 gnome-volume-manager
 4643        0 /usr/lib/bonobo-activation/bonobo-activation-server
 4648    10346 update-notifier
 4655    37128 nm-applet
 4658      128 /usr/lib/gnome-vfs-2.0/gnome-vfs-daemon
 4661     4058 gnome-cups-icon
 4680    28102 gnome-power-manager
 4718     3388 /usr/lib/nautilus-cd-burner/mapping-daemon
 4742       59 /usr/lib/gnome-applets/gweather-applet-2
 4758   290997 gnome-terminal
 4766    21571 gnome-screensaver
 4768        0 -bash
10005    12195 /usr/lib/notification-daemon/notification-daemon
11068        0 -bash
11099        0 -bash
14420        0 -bash
14503        0 -bash
14529        0 -bash
14595        0 -bash
  539        0 pstimeouts

pstimeouts has -a and -u options that do the same as their ‘ps’ counterparts. -u will show each timeout type separately.

the kernel patch (which applies against ubuntu linux-source-2.6.17-7.20) and the source for pstimeouts are located here: enjoy :)

future work: make a ‘top’ sort of utility.

vim syntax highlighting for glib, gobject, gdk and gtk types

i’ve created a file to enable vim syntax highlighting of glib, gobject, gdk and gtk types.

i handled the base gtypes (gboolean, gpointer, etc) by hand but the rest of them were generated semi-automatically using the following commands:

there are probably some inaccuracies in both directions which could be fixed with better handling but it looks to be pretty good. i considered grepping for ‘get_type’ functions in the symbols list but then you don’t get typedefs for things like callback function types.

# from inside glib source tree
find | grep .h$ | xargs cat | sed -e 'sx[/*].*$xx' |
       tr ' \t' '\n\n' | grep ^G[A-Z][A-Za-z]*$ |
       grep -v '^[A-Z]*$' | sort | uniq > ~/GTypes

# from inside gdk/ directory of gtk source tree
find | grep .h$ | xargs cat | sed -e 'sx[/*].*$xx' |
       tr ' \t' '\n\n' | grep ^Gdk[A-Z][A-Za-z]*$ |
       grep -v '^[A-Z]*$' | sort | uniq > ~/GdkTypes

# from inside gtk/ directory of gtk source tree
find | grep .h$ | xargs cat | sed -e 'sx[/*].*$xx' |
       tr ' \t' '\n\n' | grep ^Gtk[A-Z][A-Za-z]*$ |
       grep -v '^[A-Z]*$' | sort | uniq > ~/GtkTypes

# from ~
cat ~/G*Types | tr '\n' ' ' | fmt -w 60 |
  sed -e 's/^/syn keyword cType /' > gtk-highlight.vim

thanks to tpope on #vim for telling me about ~/.vim and to jdub for ‘fmt’

thoughts on summer of code

there are two or three things about summer of code that are really awesome.

number one is that you’re given a mentor who you can harass. this is fantastic. this person can function both as a helper in terms of “how do i do this in gtk?” and also in terms of “i have a tough decision to make; what do you think?”

number two is probably the most important. it’s that you’re given a mandate. i’ve written a lot of code before that i’d have hoped would make it into gnome but hasn’t. with summer of code, you’re told by the community that you’re doing something that they want done. there is a very high probability of your code appearing in a future gnome release. this is very exciting and motivating.

you’re also (or at least it feels like it) given help from the community to a level that you’re not normally given it. this could be caused by any number of factors. the people in the community might be kinder to you because you’re a summer of code student. you also may feel some solidarity with other summer of code students and discuss things with them. it might also just be that because you’re motivated by the programme in general (see point two) you’re more interested in figuring out all sorts of intricate details of how things work and therefore ask more questions.

i’m always tinkering to a certain extent and learning about gnome and gtk and gobject. i can say without a shadow of a doubt, though, that participating in summer of code has accelerated this process for me. the past little while i’ve learned an awful lot. compared to my state before the summer i feel like an elite gtk hacker.

this is important. having people in the community who understand how things work is important.

notice that in this entire post (so far) i haven’t said anything about money. i think all of the effects listed above are independent of money.

it would be really cool if we could continue to use the sort of power created by the summer of code. what follows is a sort of idea how that could possibly happen.

gnome love, bugdays, etc are all very nice, but they lack structure.

what would be really nice is a structured mentorship program. there could be some token amount of cash reward (certainly no more than $500). there could be a loose community voting process on what makes a worthwhile project. involved, knowledgeable and influential hackers in the community (like my mentor, vincent, for example) could propose to mentor projects that they feel are worthwhile. if the community approves their proposal then applications are accepted for work on this project.

it seems like a good idea to solicit applications for a few projects at the same time. having a few students hacking at the same time is important (for solidarity reasons). for a given set of applications a specific area should be targetted in such a way that the students are working on a similar area but won’t step on each others’ toes. applications in general are good because they avoid work duplication. the token prize amount would serve to offset the hassle of going through an application process and motivate people to do so.

having everyone working on a similar area is also good in terms of the next release of gnome coming out and users being able to say things like “wow! a lot of great work has been done on nautilus this release!”.

the primary goal of the program would be not to get lots of cool stuff done on gnome (although that’s nice) but rather to get people involved in the community. once you’ve done a nice chunk of work that’s released as part of gnome you feel a greater involvement in and attachment to the community. as such, the programme would not be open to people who had been involved in the programme in the past or in summer of code or wsop.

end braindump. for now.

fontcrime – less is more

in two ways, really.

i had a talk with clare the other day during the lunch break of dave parnas‘s talk at formal methods 2006.

apparently rendering of kanji on ubuntu is really awful. each character looks different from the next. a dirty secret of pango comes out.

i attended behdad’s talk about how all of this works at guadec and pango decides what glyph will be used for a specific character on a character-at-a-time basis. this means that you can easily have different fonts chosen for different characters in the same string. what’s more — pango seems to like to pick from fonts with fewer glyphs (thinking that they’re more specific and therefore better) — “less is more”.

the problem is that this is often an evil policy.

these fonts look quite different from each other. if you look at the screenshot, the right radical (“bird”) in each shot should look identical. they are very different.

sure enough, if you put these characters side by side in a string, you get one drawn in each font.

in this case, arial unicode (which i installed myself) has complete coverage of (at least nearly all) kanji. however, on a default ubuntu install you have kochi gothic which is used by preference. remove kochi gothic and you have baekmuk dotum.

if you remove both of these fonts, however, you’re left with just arial unicode (if you’re blessed enough to have this font). now things are beautiful. less is more.

i’m not up on all of this font stuff but why is it that we don’t have a big free font with all of the glyphs for every language in it? if it does exist then why isn’t ubuntu shipping it?

also — why can’t pango do a sort of prescan on the string to look at all of the characters in the string and do its hardest to try to pick a font in which all of the characters exist and use that for every character? even if there are higher quality glyphs for some of the characters in one font, it sort of seems more important that the string is displayed in a consistent font. would this be entirely too expensive?

of course, on the other side of this argument, you have the case of a single (for example) kanji character appearing inside a huge block of latin text and causing the latin to be rendered in a lower quality junk font that just happened to be inclded in the same file as the kanji…


your domain, "" is expiring

excuse my caps…

Today is August 21st — the Summer of Code deadline.

It’s been a pretty fun ride. I’ve been getting up close and personal with the gnome-panel code in ways that I wouldn’t previously have imagined to be possible.

I’ve been working on an API for applets. In addition to this API, I’ve written an implementation of a client-side library for this API and in-panel code for this library to communicate with.

The communication is done by way of DBUS and XEmbed. The API has been constructed in such a way, however, that other than GTK itself, all underlying implementation detail is invisible. If DBUS should fall out of fashion, it will be possible to replace the back-end implementation without an API shift. This is not the case for the current applet system which is very much tied to Bonobo.

Note: the API is UNSTABLE. It will change. I promise.

From the standpoint of the user (applet writer) a class is provided named GnomeApplet. You subclass this class to create your own types of applets. To give a feel for this API, I will present an example of a very simple applet.

The applet in question is a simple clock. An icon has been added for demonstration purposes.

I’ll assume for the sake of simplicity that the following function has been provided:

const char *format_current_time (gboolean seconds, gboolean _24hour);

This function returns a string representing the current time either in 24 or 12 hour format and with or without seconds.

The code of the applet is included here:

#include <libgnome-applet.h>

GNOME_APPLET_DEFINE (ClockApplet, clock_applet,
  gboolean _24hour;
  gboolean show_seconds;

static gboolean
update_time (ClockApplet *clock)
  gnome_applet_set_text (GNOME_APPLET (clock),
    format_current_time (clock->show_seconds, clock->_24hour));
  return TRUE;

static void
update_boolean (GnomeApplet *applet, const char *key,
                gboolean value, gpointer user_data)
  *(gboolean *) user_data = value;
  update_time ((ClockApplet *) applet);

static void
clock_applet_initialise (ClockApplet *clock)
  GnomeApplet *applet = GNOME_APPLET (clock);

  gnome_applet_set_name (applet, "Clock Applet");
  gnome_applet_set_persist (applet, TRUE);

  gnome_applet_add_dropdown_widget (applet, gtk_calendar_new ());
  gnome_applet_set_image_from_stock (applet, GTK_STOCK_YES);

  gtk_timeout_add (1000, (GtkFunction) update_time, clock);
  gnome_applet_config_watch_boolean (applet, "24hour", update_boolean,
                                     &clock->_24hour, NULL);
  gnome_applet_config_watch_boolean (applet, "24hour", update_boolean,
                                     &clock->show_seconds, NULL);

The first thing to notice is the GNOME_APPLET_DEFINE line. In its simple form, this line is like a G_DEFINE_TYPE instantiation minus the parent class (since the parent class is always GnomeApplet).

For example:

GNOME_APPLET_DEFINE (ClockApplet, clock_applet);

However, this macro aims to reduce your typing for you. As such, you do not need to define structures for your class. If you want additional items in these structures (other than the parent class) then you give them using the syntax seen in the example code.

The update_time function is uninteresting. It shows that the fields defined in the extended syntax of GNOME_APPLET_DEFINE are accessed as you’d expect them to be. It also shows use of the gnome_applet_set_text function.

The new applet API is a hybrid API. As a subclass of GtkContainer, you can gtk_container_add a widget to the applet and it will be displayed on the panel. You are much more likely, however, to want to use the set_text and set_image functions.

The first time you use a set_text or set_image function the applet will internally create a GnomeAppletLayoutBox and add it to itself. It will then request the creation of the image or label inside of this box.

The GnomeAppletLayoutBox system allows you to easily create applets that feature 0 or 1 icons and 0 or 1 text labels. In addition, it gracefully handles horizontal and vertical panels and drawers of varying thickness.

The label that is created is not a GtkLabel, but rather a GnomeAppletLabel. GnomeAppletLabel has some functions that make it easier to use than a simple GtkLabel for purposes of putting in a GnomeAppletLayoutBox. The most noticeable feature of a GnomeAppletLabel, however, is that when placed on a coloured, pixmap or transparent panel, it shows a drop shadow like Nautilus shows on its desktop contents.

update_boolean is not a very interesting function either. It simply updates a boolean value pointed at by the user_data parameter and calls update_clock. The need for functions like this may disappear as the API becomes more developed.

clock_applet_initialise is the most important function here. When you use GNOME_APPLET_DEFINE the 2nd parameter has _initialise added to it and this is the name of the only function that you must implement.

This function is NOT a normal GObject init function. It would be called clock_applet_init if that were the case. This function is, rather, called after the object has been entirely constructed and connected to the panel. This means that you can perform function calls from it which query the state of the panel.

We perform a number of calls here:

gnome_applet_set_name tells the panel the name of the applet. This is used in UI that the user sees concerning the applet. For example, of the applet were to unexpectedly exit, the message dialog shown would be customised with the applet’s name.

gnome_applet_set_persist registers the applet as a persistent applet. When an applet is first created it is non-persistent. This means that the applet can come and go as it pleases. The panel will make no attempt to start the applet on login and won’t complain if the applet just vanishes. This behaviour is useful for programs like Gossip or Muine, where the life of the applet is tied to the life of the application, rather than the life of the session. It is also useful for applets that wish to come and go as hardware is added or removed.

If an applet is marked persistent then its current path is noted down in the panel’s GConf tree. On login the applet will be invoked and asked to join the panel at its previous location.

gnome_applet_add_dropdown_widget is a new feature. When passed a GtkWidget, this function will add the widget to a dropdown display. Many applets would or currently do benefit from this functionality. For the clock, it makes sense to have a dropdown calendar. For the weather applet you could have a dropdown detailed forecast. For the mixer, a dropdown slider. If a dropdown is registered then it is displayed when the left mouse button is clicked. If the user moves their mouse from applet to applet while a dropdown is displayed then the dropdowns of the different applets are displayed.

gnome_applet_set_image_from_stock sets the image shown with an applet. There are functions for setting from stock, from file, from icon theme, from pixmaps and from pixbufs. See the discussion above about GnomeAppletLayoutBox.

gtk_timeout_add is gtk_timeout_add.

gnome_applet_config_watch_boolean is showing a very small part of the configuration system that exists. It was decided, for a number of reasons, that applets would not link against GConf. It is still possible to do this for yourself and use GConf functions with your applet, but this behaviour is not recommended. All configuration requests are, instead, sent to the panel via DBUS.

There are a number of reasons for this:

  1. The configuration information associated with an applet should be tied to the panel (as it is now). The panel might be using a different configuration system then the applet. (think: mixing KDE, xfce, GNOME).
  2. If GNOME itself shifts configuration systems, the GConf API is completely hidden. (think: you won’t need to rewrite your applet).
  3. There is a memory savings associated with not linking to GConf in every applet, not firing up the machinery in every applet and not having a server-side connection for every applet.
  4. There is an additional benefit interms of load time. Since the panel can preload all of the information for all of the applets in a single roundtrip to the GConf daemon, the number of roundtrips is reduced. This means less contention for a heavily contended resource.

There are a few unfortunate effects:

  1. You can not store pairs or lists. Only values which fit in a GValue are supported. I could add more API, but the point is to not expose GConf-specifics.
  2. I had to do a lot more typing. More typing introduces the possibility of more bugs and certainly, the interface won’t share the exact semantics of GConf.

In general, there are 5 flavours of each function. One for each of GValue, string, int, boolean, double. I’ll introduce the integer variants:

  • gnome_applet_config_get_int – gets a key of a given name and returns it in a pass-by-reference variable. This function returns TRUE if the key existed or FALSE if it did not.
  • gnome_applet_config_get_int_default – gets a key of a given name and returns it. If the key did not exist then a user-provided default value is returned.
  • gnome_applet_config_set_int – sets a key of a given name to a given value. Returns TRUE if successful.
  • gnome_applet_config_watch_int – registers a watch function on a given key. A user_data (with corresponding destroy notify) field is also given. When this function is first called, a synthetic notify event is immediately generated. You can, therefore, use this function to initialise variables without having to manually call one of the ‘get’ functions.

There is more API than shown here, but this should give a good overview of the general flavour. This project is not yet done! The following features are planned to be completed before GNOME 2.18 when, if all goes well, this code will ship as part of gnome-panel.

Immediate TODOs:

  • gtk-doc for the client-side API with the documentation posted on
  • integration of Vytautas’ work to make multiple instances of the same applet use the same process
  • support for dynamic loading of applets into the panel to further reduce memory use
  • improved support for finding and loading persistent applets (the path for invoking applets on startup is currently hard-coded to my working directory)

Other TODOs:

  • gather feedback from early adopters of the API. What needs to change? Can there be some more convenience functions?
  • look into language bindings (python and C# are likely early candidates)
  • bug fixes! I can’t even imagine how many bugs are in the code. It will take time to reveal them all and get the code to be stable enough for general use.

A recent version of the code is usually available at I’m going to wait until after the GNOME CVS branches 2.16 stable until I commit the code to CVS HEAD.

burning cpu and battery on the gnome desktop

last night seb128 asked me for advice on how he could tell if gnome-vfs was using inotify or gamin as its backend for file monitoring. i didn’t really know, so i suggested stracing gnomevfs-monitor.

this is when i found out that the inotify code in dapper’s gnome-vfs wakes up 10 times a second. i understand this to be because it does some polling for non-existant files and also has an internal timer to match move_from and move_to events.

this is a problem. 9 applications on my system right now are using file monitoring and suffering this problem. list is as follows:

  • panel
  • nautilus
  • settings-daemon
  • ephy
  • update-notify
  • vfs-daemon
  • seahorse-agent
  • muine
  • deskbar

this alone causes my system to wake up 90 times per second.

at guadec i also talked to mclasen about another bug. gtk in dapper wakes up whenever a mouse button is clicked or a modifier key is pressed. this means that every single application on your desktop wakes up every time you press or release shift. ow. mclasen fixed this bug within a very impressive amount of time and the fix will be in gtk 2.10. dapper users are left burning cpu for the time being.

i’ve been going around stracing random pid’s of desktop processes. i’ve found some other offenders.

  • gnome-power-manager wakes up twice per second to do something
  • battery applet wakes up once per second to do something
  • clock-applet wakes up once per second to update the time even when seconds aren’t shown
  • gajim wakes up 10+ times per second for some unknown reason
  • at-spi-registryd wakes up more like 20+ times a second (?!?)
  • gss seems to talk to x11 once per second (presumably to ask if anything has happened). i don’t understand why it has to do this so often.

so all in all gnome is causing my computer to wake up about 200 times a second when totally idle. since this is happening in a lot of different processes, that’s one heck of a lot of context switches.

i did a check. in a loop (with sleep 10s) i catted the contents of the acpi battery state in /proc. i did this with gnome running and with a failsafe x session with xterm. gnome causes my laptop to use a couple of watts more power.

i also was talking to mjg59 about this earlier today. according to him, circa 2.6.19 we’ll see a new kernel feature by which the ‘hz’ timer that runs at a constant rate will be disabled. the idea is that we can use the new hightech timers we have to schedule timer interrupts only when needed. this means that except when in userspace (for preemption purposes) or waiting for some specific event the timer will be disabled and the cpu will be totally idle until the next irq comes in.

if we can reduce the number of wakes-up the desktop does (to say, zero times per second) then we can reduce the number of times the kernel has to wake the cpu up at all. even without the new timer tech there’s a noticable (albiet small) change in battery consumption. with the new timers, not waking up will become increasingly important.

this post is a plea. please fix your apps to never wake up unless the user does something to them. my laptop’s battery will thank you.

ps: the best way to find out how often your app is running is to use strace. you have two options here. you can either run your application directly under strace as follows:

strace yourapp

or you can attach strace to a running application by its pid:

strace -p `pidof yourapp`

if you see poll( or select( and a long wait then your app is sleeping (good!). if you see lots of activity when you’re not otherwise interacting with your application then you’ve got trouble. fix it!

macbook sleep now working

yesterday, starting with some pointers from mjg59, i tried to find the cause of my macbook crashing on wakeup from sleep.

i narrowed it down to the code which resumes the pci bridge. as it turns out, the kernel restores the pci configuration space by copying in the values it had before the suspend. it does this in a bad order though, and the card becomes enabled before it is fully configured.

reversing the order fixed the problem — sleep now works on the macbook.

benc has merged the fix into the dapper git tree and the changes will be available with the next dapper kernel release.

there are still two small problems (one with a very easy fix). the harder once is acpi. when returning from sleep a random irq9 is thrown with no handler registered causing all further irq9s to be disabled. this means no acpi events get reported. no idea how i’d start fixing this. please help.

macbook backlight control

today i got sick of my macbook backlight brightness controls not working in dapper and set out to do something about it. while sitting in ghazi’s with my good friend wolfgang, i set out to find and reverse engineer the apple kernel extension responsible for controlling the backlight brightness.

the result is that i have written an application to control the backlight on the macbook in linux.

the ubuntu dapper on the apple macbook page has been updated accordingly.