today’s hack: pstimeouts

today i wrote a kernel patch to find out just how often processes are waking up as a result of setting timers for themselves. the intention is to provide a tool to make it extremely easy to spot poorly behaved applications.

the kernel patch recognises 5 types of “timers” that might cause a process to wake up:

  • poll returning due to timeout after sleeping
  • select, same as above
  • epoll, same as above
  • the ‘real time’ interval timer (SIGALRM)
  • anything else inside the kernel that uses schedule_timeout

the kernel patch makes this information available in /proc/pid/timeouts as 5 numbers (corresponding to the list above).

i wrote ‘pstimeouts’ as a small utility to read this data and present it to the user. it’s written in straight-up C with no library dependencies (not even glib) as to be usable by the biggest number of people.

here is a ‘screenshot':

desrt@acquiesce:~$ pstimeouts
  pid timeouts process
 4568       44 x-session-manager
 4607        7 /usr/bin/dbus-daemon
 4609      704 /usr/lib/libgconf2-4/gconfd-2
 4612        0 /usr/bin/gnome-keyring-daemon
 4615     6615 /usr/lib/control-center/gnome-settings-daemon
 4629     5428 /usr/bin/metacity
 4634    39574 gnome-panel
 4636    17635 nautilus
 4639     1391 gnome-volume-manager
 4643        0 /usr/lib/bonobo-activation/bonobo-activation-server
 4648    10346 update-notifier
 4655    37128 nm-applet
 4658      128 /usr/lib/gnome-vfs-2.0/gnome-vfs-daemon
 4661     4058 gnome-cups-icon
 4680    28102 gnome-power-manager
 4718     3388 /usr/lib/nautilus-cd-burner/mapping-daemon
 4742       59 /usr/lib/gnome-applets/gweather-applet-2
 4758   290997 gnome-terminal
 4766    21571 gnome-screensaver
 4768        0 -bash
10005    12195 /usr/lib/notification-daemon/notification-daemon
11068        0 -bash
11099        0 -bash
14420        0 -bash
14503        0 -bash
14529        0 -bash
14595        0 -bash
  539        0 pstimeouts

pstimeouts has -a and -u options that do the same as their ‘ps’ counterparts. -u will show each timeout type separately.

the kernel patch (which applies against ubuntu linux-source-2.6.17-7.20) and the source for pstimeouts are located here: http://desrt.mcmaster.ca/code/pstimeouts/. enjoy :)

future work: make a ‘top’ sort of utility.

9 Comments

  1. Posted September 18, 2006 at 10:43 am | Permalink

    Cool. I assume those numbers are cumulative since last boot?

  2. desrt
    Posted September 18, 2006 at 10:51 am | Permalink

    Since the process was created with fork().

  3. Nicholas Miell
    Posted September 18, 2006 at 11:56 am | Permalink

    Hate to tell you this, but you’ve just reinvented the wheel:

    http://www.ussg.iu.edu/hypermail/linux/kernel/0508.1/0106.html

  4. desrt
    Posted September 18, 2006 at 12:19 pm | Permalink

    i looked at timertop and ran it. it only tracked when high precision timers caused wakeups.

    this patch tracks things like poll/select as well which is a lot more useful for the specific use case of finding gtk applications that schedule timers to run more often.

  5. Ken
    Posted September 18, 2006 at 1:57 pm | Permalink

    You, sir, are a god.

    A minor god, to be sure — somewhere between Pellor and Mnemosyne, I’d say — but still quite bodacious.

  6. Posted September 18, 2006 at 5:45 pm | Permalink

    Looks good Ryan. Keep up the good work.

  7. Timothée Lecomte
    Posted September 18, 2006 at 10:32 pm | Permalink

    I appreciate this kind of work.

    I have a question though. I wrote a peace of code for gnuplot where I have to care about stdin from the command-line interface and at the same time to look for the GUI events (clicks, moves, keys) coming from the ‘plot’ window. The main thread is a loop on a function called waitforinput() which is supposed to return the next character from stdin while processing the GUI events when they arrive. There is a second thread which is a usual GUI loop, and when it processes an event, it forwards it to the main one using a mutex’ed list. I ended up using select() on stdin with a timeout to periodically check for the contents of this list. So gnuplot falls in your category of “bad programs”, but it’s because I don’t know how else to do ! Do you have an idea ?

  8. desrt
    Posted September 19, 2006 at 2:10 am | Permalink

    Timothée: the only real solution here is to use a single mainloop for X events and your stdin.

    Toolkits like GTK will let you register a fd watch (like for stdin).

    Otherwise, you can get the X socket fd from the display using the ConnectionNumber() call and add it to your stdin mainloop that way (this is what GDK does internally).

  9. Timothée Lecomte
    Posted September 22, 2006 at 1:37 am | Permalink

    desrt: thanks !