docs hackfest in Toronto

as i’m writing, the last of the docs hackers are leaving toronto. from what i can tell, they had an extremely productive week. from my perspective it was sort of fun to act as the on-the-ground guy for a change. i also managed to write a patch or two against yelp, so my attendance at the hackfest wasn’t entirely symbolic…

the hackfest was hosted by CDOT (the centre for development of open technology) at seneca college in (very) north toronto. these guys are really cool — a bunch of fedora and mozilla hackers as an official department of a college. a big thanks to anyone there who is reading this post; it was really awesome to hold a hackfest in such a cool place.

a step back

(standard junk: this is my personal opinion and i’m possibly ethically compromised because i’m currently on contract working for canonical, etc. etc. blah)

canonical does a lot of things that i would classify as pretty boneheaded in terms of their relationship to various free software communities. they have an interesting and colourful history with quite a lot of projects and our project is pretty close to the top of that list.

it’s my opinion that canonical takes a more pragmatic approach than most free software projects have. they have a bit more of a “…and damn the consequences” attitude. they’ve made a lot of decisions that have put them at odds with a lot of people. i’ve found myself on both sides — defending their choices when i agree and calling them out when i don’t.

binary drivers so that it “just works”? win. copyright assignment? not such a win. this mess with banshee? ya. that’s pretty lame.

i’m sure everyone can think of a few more “situations” off of the top of their heads.

canonical gets a lot of flack around these parts, and rightly so. they often make decisions that leave a lot of us scratching our heads and wondering why. they need to be called out. i’m glad to see it happening. i’d do more of it myself, but actually i hate writing blog posts.

to some casual readers of planet lately, it might seem as though the opinion of canonical in the gnome community is quite negative. i think that even those deeply involved in our community, in heated moments, get pretty pissed off.

taking a step back though, i think that it’s clear that just about every poster here would agree on one thing: that canonical is a net positive to the world of free software and that they are helping us achieve our most important goals.

as a fun thought experiment, imagine if even the worst dreams came true: next year canonical takes their copyright ownership of their qt rewrite of unity and makes it closed source to make lots of money selling it on embedded devices. imagine one of these devices is actually awesome and reasonably priced. i can tell you one thing about that: i’d be the first in line to get my hands on this device. i’d be excited as hell about it. a few closed components on an otherwise totally open os sounds pretty good to me. better than, say… android (that everyone seems to love so much). pretty comparable to meego (which everyone loved even more until quite recently).

now snap back to reality and remember that these worst dreams are just dreams and actually canonical is actually even better than that. there’s no closed source component at all. they’re actually taking a pretty high road with just about everything that they’ve done so far.

when you think about the amount of slack that we’ve cut companies like google and nokia and stack it up against the amount of condemnation that we’ve seen hurled at canonical it becomes easy to forget one thing that i think most of us would agree on: canonical is a very close friend.

the paradox is true: you save your strongest criticisms for those you love most.

more on dconf performance, btrfs and fsync

dconf performance

i’ve been working on dconf recently. one of the things i’ve been up to is writing some proper test cases. of course, testing is a good chance to check performance…

the test case is written against the dconf gsettings backend. it generates 1000 random changesets and writes them into an empty dconf database while tracking the changes in two hash tables. one hash table is managed explicitly by adding all changes to it and the other is managed implicitly by connecting to change notification signals and updating itself when they are received. after each of the 1000 changesets is written a full three-way comparison is performed to ensure that the tables match each other and the contents of the dconf database.

the test system is a quad core i7 at 2.67GHz.

while the 1000 writes are going on the performance is quite awful. each dconf lookup takes approximately 30µs (compared to approximately 0.16µ for a GHashTable lookup, for comparison). that’s about 200 times slower. this can be almost entirely blamed on the fact that the gsettings keeps a local list of outstanding changes (those that have not made it to disk yet) and scans that list linearly on each read. you won’t run into this performance case unless your app is doing a *lot* of writes all at once.

the test then waits for the writes to make it to disk and runs the three-way comparison 1000 times over again. the results of this test are a more fair indication of dconf read performance under “normal” conditions. it takes approximately 1µs to do a lookup using the dconf gsettings backend (which is approximately 7 times as long as it takes to do a GHashTable lookup). for some time i’ve been telling people that “dconf lookups should be within an order of magnitude of GHashTable performance” and now i have hard numbers for that. this also gives a lower bound for GVDB performance (since dconf uses it).

tl;dr: a dconf read takes 1µs (which is approximately 7 times GHashTable).

btrfs and fsync

when doing this testing i noticed that dconf is really slow. it takes 64ms for the dconf-service to write each change to the disk. the reason for this is because it does an fsync() after each and every change. i have spinning disks. i sort of expected this and i designed around it; that’s why the gsettings backend for dconf keeps the local list of changes in flight: your app doesn’t wait for this 64milliseconds (and is actually completely unaware of it). it starts to look bad, though, when you’re doing 1000 writes.

i did a bit of reading and i discovered something very cool: btrfs guarantees that overwrite-by-rename is atomic. no need to call fsync() between writing the data to the temp file and renaming it over the old file. i have a patch to glib in #637544 to detect when g_file_set_contents() is being used on btrfs and skip the fsync() in that case. this reduces the amount of time taken to write a change from 64ms to 3.5ms.

tl;dr: btrfs is cool.

edit: another interesting dconf statistic: the dconf database generated by one instance of the test case is 475680 bytes on disk while containing 4455 entries. that’s about 107 bytes per entry.

edit2: after applying the glib btrfs no-fsync patch the 30µs dconf lookup time in the first phase of the test drops to about 4µs per lookup. this can be attributed to the fact that the service is having a much easier time keeping up with the write requests and therefore the list of outstanding changes (which is linearly searched) is kept much smaller.

gsettings is fast

sometimes people will come up to me at a conference and one way or another mention that they are avoiding using gsettings because they need their app to start “really fast”. at uds for example, someone asked me “i should be using a keyfile for this, right?”.

gsettings has dconf as its backend. there are a couple of things that i assumed were common knowledge about dconf that surprised people when i told them. the main two things to note are that the on-disk dconf database is a hashtable that gets mmaped into your process and that reads (after the first read) typically involve zero system calls — just direct access to the hash table.

i still decided that it would be helpful to get a hold on some actual numbers here, though. i did some testing (nothing serious — but it gives a good idea of the ballpark figures involved here).

my methodology for measuring how long it takes to do something is this:

time (for i in `seq `1000` do; ./something; done > /dev/null) and dividing the ‘real’ time by 1000

running ‘/bin/true’ takes about 1.1ms
running a do-nothing program linked against libgio and calling g_type_init(): 2.2ms

when i went to benchmark gsettings i noticed that it was a bit slower than i thought. about 9ms to run the gsettings command line tool to “get” a setting. (for comparison, initialising gtk is 30ms and qt 350ms). still, i was wondering why it was so slow. it turns out that the largest part of that was that i was blocking on gdbus initialisation which (due to the chatty nature of the dbus protocol initialisation and the fact that dbus-daemon is a slow talker) takes quite a long time. gdbus needs to be initialised along with gsettings in order to add match rules for change notification.

i’ve fixed the backend so that we don’t block on gdbus initialisation anymore — it happens asynchronously and in another thread. those changes will land on master today. after the changes, running the gsettings commandline tool in ‘get’ mode takes about 4.2ms.

so for the record: the cost of initialising gsettings and reading a value out of dconf adds about 2ms to your program startup — less than 1/10th of the time it takes to initialise gtk and on the same order as the length of time it takes to spawn the process, load the shared libraries and call g_type_init().

gtk hackfest summary

the gtk hackfest came to a conclusion a bit over a week ago. since then we’ve had a gtk team irc meeting with the release team present to discuss the results.

first, i’d like to thank everyone who came to the hackfest for the awesome work done. also, thanks to the gnome foundation for paying for the hotel and to the companies involved for sending their people (thanks to codethink in my case). thanks also to codethink for sponsoring the catering and lunches and lanedo for the “official dinner” of the hackfest. a very big thanks goes to igalia for the hosting and particularly to alberto garcia for being a completely awesome host — seriously, you rock.

the biggest news out of the hackfest is that we have a very solid roadmap for the future of glib in gtk in two ways. first, the path to gtk3 is very clear at this point. we have an ambitious but realistic feature list for what we want to see added in the next two months and each feature on that list has a name assigned to it. it’s a heck of a lot of work, but the release is currently on schedule and things are proceeding quite smoothly.

second, we have a list of things that we want to take care of for a future glib4/gtk4 release.

both of those lists are in a rough form here: http://live.gnome.org/Hackfests/GTK2010/RoadmapDiscussion

the roadmap is going to be polished and put into a more user-centric form at the upcoming boston summit (this weekend).

perhaps the most surprising takeaway from the hackfest is that gtk4 is coming quite soon. we plan to do the bulk of the work required to get it out the door in 2011. we were originally saying that we’ll target the gtk 4.0 release for “december 2011” but we’ve decided that it’s probably a little bit too much to say that and rather we will say that we want a feature-complete beta release by that time. of course, deadlines being as they are, who knows what will happen….

we plan to open up a gtk4 branch as soon as gtk 3.0 is out the door. we also plan to have another gtk hackfest in the spring of 2011, and we may find ourselves wanting one again in the fall.

having two major releases with so little time between might seem very weird. we tried for quite some time to consider reasonable alternatives (which is part of why this blog post is so far delayed). the decision here comes down to two simple facts. first, gtk 3.0 is required for gnome 3.0 and was promised for christmas. second is that we have an awful lot of well-directed momentum at the moment and we have a lot of really large changes that we want to make that won’t fit into a 2 month window.

something also worth mentioning, for those who didn’t know, is that samsung sent 3 engineers to the hackfest. for some time now, they’ve had a team of hackers (somewhere in the area of 20-30 strong) working on a phone based on gtk, to be released soon. they had a pretty slick prototype present and brought some interesting viewpoints to the discussion. boram park (one of the samsung engineers) also contributed samsung’s first “direct” upstream patch to gtk during the hackfest.

as for myself, my personal focus on gtk3 and glib 2.28 for the next two months falls into two large tasks. one: fix time in glib and get GPeriodic spiffed up and make sure that it’s properly wired up as a paint clock for gtk. two: get GtkApplication rocking with some nice features like full shell integration, automatic session management and the like. i’m particularly interested in feedback from early adopters about ways in which i could make your life easier.

i’m going to be at the boston summit again this year, and there will surely be a lot of discussion there on getting the gtk3 story closed.

GNOME_ME_HARDER

this week i’m at uds with vuntz and andreas.

there was obviously a big announcement yesterday. enough people are commenting on that already, and i don’t have anything that i really want to add to that conversation.

on a more positive note, i want to mention something that was agreed during one of the desktop sessions yesterday morning.

based on a promise made by jono bacon that gnome-shell would be an absolutely first class citizen in ubuntu (just not included on the CD) and in response to the lessons learnt from the failure of “stracciatella gnome”, we proposed the creation of an environment variable. if set, it disables the effects of any ubuntu vendor patch to upstream components when those effects are related to gnome-shell vs. unity integration.

what this means is that you will be able to login to a gnome-shell ubuntu experience that works properly, without all of the applications assuming that they are trying to interact with unity.

i like the name GNOME_ME_HARDER, but i guess they’ll probably want to do something a little more boring.

the specific parameters of the promise that was made are as follows:

1. for new vendor patches, this “works with shell” functionality will be required in order for the patch to be considered functionally complete for inclusion.

2. for existing vendor patches, missing GNOME_ME_HARDER functionality will be considered as a bug. this means that issue will receive the same treatment as other bugs — it will be fixed if there is time, patches welcome, etc.

gtk hackfest, day 2

codethink sent me to the gtk+ hackfest this week, hosted in the igalia offices. some really nice things are happening there so far.

first of all, i landed GApplication on glib master (and in gtk too). sorry for any breakage that may have caused. actions support should be coming soon…

second, after some discussion with emmanuele, i sent out to write a paint clock that could be shared by both gtk and clutter. i stayed up until 4am last night dumping my brain into vim and the result of that work is in glib (gio) in the form of GPeriodic. there is also a branch in gtk (wip/paint-clock) that uses GPeriodic to schedule repainting of windows and (for example and proof of concept) to animate GtkSpinner. presently cody is working on a somewhat less-trivial case: making GtkExpander animate smoothly. emmanuele is currently attempting to rebase clutter’s master clock on top of it, so the end result is that clutter and gtk should be on the same clock by the end of the week.

there is not currently any support for merging of input events. that’s mostly due to the fact that it’s a really hard problem and we probably won’t get a chance to deal with it at all before 3.0.

the other thing that we’re working on this week is reducing the GdkWindow abuse that occurs in gtk. samsung sent a few hackers to the hackfest, and one of them (boram) is trying to replace the 2+n input/output GdkWindows in GtkEntry with input-only windows. we believe that this is a reasonable intermediate step to improve use of the new gtk_widget_draw() api without having to think about the input issues (again, because they are quite hard).

i’m personally about to set out on a GtkViewport reimplementation that, first, behaves very naively by always redrawing everything and later caches the content of the scrolled area in an offscreen window (including some of the non-visible areas as a sort of pre-fetching).

there is some talk that gtk4 may be coming in the nearer future than any of us anticipated……

GSettings update

after quite some time, the GVariant, GSettings, dconf saga is coming to an end. for quite some time Codethink has been sponsoring me to spend a fair bit of time on this stuff and i’m happy to say that it’s reaching the point of usability.

first of all: all existing code is now merged to its final destination. no weird branches or anything are required to use the latest stuff. in fact, i made a pair of tarball releases yesterday: one for glib and one for dconf. those are all you need.

install instructions are something like this:

1) install glib
2) install dconf
3) profit!!

i hacked together a quick example of what #3 might look like. check it out on git.desrt.ca (not in gnome git due to the fact that it will probably bit-rot soon).

a few notes with respect to the recent releases:

1) the m4 macro for GSettings has changed. if you’re an early adopter this will burn you and i am sorry. the new macro is much nicer, though. it’s less typing for you and it fixes a few problems with the old macro.

to use it, you just need to have two lines in your Makefile.am:

gsettings_SCHEMAS = one.gschema.xml two.gscema.xml ...
@GSETTINGS_RULES@

make sure your ACLOCAL_FLAGS is setup properly if you’re using glib installed in a strange prefix because otherwise aclocal won’t find gsettings.m4.

2) the dconf release as of yesterday has a small bug with an extremely large impact: the gio-querymodule support is broken which causes the dconf GSettings backend not to be seen if you run the gio-querymodule tool. this has been fixed in git and will be in the next release (which should be along in a few days) and it only affects you if you run the tool. my understanding is that vendor packages are working around the issue already.

vendor packages are coming for fedora (rawhide) and ubuntu (maverick, plus backported to lucid in an official ppa). your least-pain option is probably to wait for these.

the next release of dconf will be focused on using it outside of GSettings; a standalone client-side library, and a commandline tool. Robert Ancell has also signed on to do some work on a graphical editor.

one last note: dconf is currently extremely slow. “hilariously slow” might be the appropriate thing to say. don’t worry — it’s just a small fix to make that better. for now i’m more interested in features work, though.

one more last note: the dconf file format might change. just so you know. :)