Benchmarking Login Time Improvements

I finally got around to committing some changes to GConf which Lorenzo Colitti’s analysis of GNOME startup time showed would make a considerable improvement to login time. More details here.

The most frustrating part of this is just how difficult it is to get decent, reproducible figures of login time to show the improvements.

I first tried using bootchart to reproduce the results as Lorenzo had done. I hacked up a script to run gdm on a test machine, automatically login as a test user and get a bootchart of the time between gdm starting and the panel finishing loading the applets. I did a number of runs for each of my test cases – unmodified GConf, unmerged tree; unmodified GConf, merged tree; patched GConf, merged tree.

Unmodified Merged Tree Merged Tree, Split Translations

You can see improvements, just as Lorenzo predicted, but annoyingly each run of bootchart varied wildly from others, so you can’t really be sure exactly what difference you’ve made.

Next, I tried a much more low-tech approach. I wrote a script to record a timestamp in a text file in /tmp and ran that script from xinitrc and from the panel (once it had finished loading applets). By making the script reboot once it had recorded the second timestamp, and configuring GDM to login automatically, I could walk away from the machine and let it record a bunch of timings rather than sit there and watch it reboot.

Disappointingly, these results varied quite a bit too. But, by discarding obvious anomalies (note my ignorance of statistical methods) and taking an average, I got:

  • Unmodified GConf – 14.382s
  • Unmodified GConf; merged tree – 13.978s
  • Patched GConf; merged tree – 11.086s

That shows a ~20% improvement in login time, which is certainly promising.

2 Responses to “Benchmarking Login Time Improvements”

  1. Regarding the problems you had with differing bootchart results — are those differences in overall time or the graph structure?

    In any case, I’m glad bootchart is being used for application profiling as well and I’d be interested in ironing out these bugs. Feel free to send bug reports :)

    Thanks,

    Ziga

  2. Eric Anholt says:

    If you want to make statistics but don’t want to actually remember how to do them, or to write a custom program to do them, then I recommend building ministat from FreeBSD’s src tree. It should be reasonably portable.

    http://www.freebsd.org/cgi/cvsweb.cgi/src/tools/tools/ministat/

    Just drop a column of all your numbers in two (or more) files, one for each test case, run ministat with them as an argument, and have math tell you whether your numbers have a significant difference.

    It’s a daily task for me, I find, and I love the tool.