Yo dawg! I herd you like browsers, so i put a browser in your browser so you can browse while you browse.
Category: General
Gtk3 vs HTML5
The last few weeks I’ve been working on an interesting new idea, hacking out a prototype.
The code is not really clean enough for public consumption yet, and a bunch of features are missing. However, its now at the stage where it can be demoed and evaluated.
I think the best way to introduce it is via a video: (original theora file)
[vimeo width=”763″ height=”512″]http://vimeo.com/17132064[/vimeo]
Basically, its a backend for Gtk+ 3 that renders in a browser.
A more techincal description for the web geeks among us:
Each toplevel window is mapped to a canvas element, and the content in the windows is updated by streaming commands over a multipart/x-mixed-replace XMLHttpRequest that uses gzip Content-Encoding to compress the data. Window data is pushed as region copies (for scrolling) and image diffs. Images are sent as data: uris of uncompressed png data.
Input is gathered via dom events and sent to the server using websockets.
Right now this is Firefox 4 only, but it could be made to work in any browser with websockets.
Now, I want to know, Is this useful?
There are two basic ways to use this, you can either run your own apps on your own server and access it from anywhere (kinda like screen). Or you can put it on a public server that spawns a new instance of the app for every user (gimp on a webpage!).
If you had this technology, what cool stuff would you do with it? What apps would you run, and how would you use them?
Tracing glib
I saw a blog entry where Krishnan Parthasarathi at sun played with static dtrace probes in glib and it seemed like a really nice thing. Also, the systemtap developers at Red Hat continue to kick ass, and I’ve long wanted to play around with their work. This seemed like a great thing to try it out on.
I’m running Fedora 12 here, and it includes systemtap 1.0 and utrace, so it support the same kind of static probes that dtrace does (in fact, the markers are source compatible with dtrace). So, I wrote an initial set of dtrace/systemtap probes and a systemtap tapset (a library for easier using the probes in systemtap) for glib and gobject.
Its kind of limited atm, but it lets you trace memory allocation in glib, gobject lifecycle issues and signal emissions. Already this lets you do some pretty cool things, like this alive.stp:
global alive
probe gobject.object_new {
alive[type]++
}
probe gobject.object_finalize {
alive[type]--
}
probe end {
printf ("Alive objects: \n")
foreach (a in alive) {
if (alive[a] > 0)
printf ("%d\t%s\n", alive[a], a)
}
}
Which you can run like this:
stap alive.stp -c "gedit /etc/passwd"
Giving something that starts:
Alive objects: 72 GParamObject 1 GdkDisplayManager 1 GdkDisplayX11 7 GParamPointer 17 GParamDouble 1 GdkScreenX11 ...
Another example is signals.stp which is a simple example to look at signal emissions:
probe gobject.signal_emit {
printf("%s --> %p[%s]::%s\n", thread_indent(1), object, type, signal);
}
probe gobject.signal_emit_end {
printf("%s <-- %p[%s]::%s\n", thread_indent(-1), object, type, signal);
}
Some example output (again from gedit), shows how size allocation works for the main window:
14406 gedit(7492): --> 0x00000000021c2040[GeditWindow]::realize 14862 gedit(7492): <-- 0x00000000021c2040[GeditWindow]::realize 14872 gedit(7492): --> 0x00000000021c2040[GeditWindow]::check-resize 14881 gedit(7492): --> 0x00000000021c2040[GeditWindow]::size-request 14890 gedit(7492): --> 0x00000000021a2950[GtkVBox]::size-request 14899 gedit(7492): --> 0x000000000236f090[GtkHPaned]::size-request 14907 gedit(7492): --> 0x000000000236f190[GtkVPaned]::size-request 14915 gedit(7492): --> 0x0000000002378010[GeditNotebook]::size-request 14927 gedit(7492): --> 0x000000000235ac30[GeditTab]::size-request 14935 gedit(7492): --> 0x00000000023b79f0[GtkScrolledWindow]::size-request 14944 gedit(7492): --> 0x00000000023825c0[GtkHScrollbar]::size-request 14954 gedit(7492): <-- 0x00000000023825c0[GtkHScrollbar]::size-request 14963 gedit(7492): --> 0x0000000002382730[GtkVScrollbar]::size-request 14973 gedit(7492): <-- 0x0000000002382730[GtkVScrollbar]::size-request 14980 gedit(7492): <-- 0x00000000023b79f0[GtkScrolledWindow]::size-request 14987 gedit(7492): <-- 0x000000000235ac30[GeditTab]::size-request 14995 gedit(7492): --> 0x00000000023b9df0[GtkHBox]::size-request 15002 gedit(7492): <-- 0x00000000023b9df0[GtkHBox]::size-request 15009 gedit(7492): <-- 0x0000000002378010[GeditNotebook]::size-request 15015 gedit(7492): <-- 0x000000000236f190[GtkVPaned]::size-request 15021 gedit(7492): <-- 0x000000000236f090[GtkHPaned]::size-request 15028 gedit(7492): <-- 0x00000000021a2950[GtkVBox]::size-request 15034 gedit(7492): <-- 0x00000000021c2040[GeditWindow]::size-request 15044 gedit(7492): --> 0x00000000021c2040[GeditWindow]::size-allocate 15051 gedit(7492): --> 0x00000000021a2950[GtkVBox]::size-allocate 15060 gedit(7492): --> 0x000000000236f090[GtkHPaned]::size-allocate 15067 gedit(7492): --> 0x000000000236f190[GtkVPaned]::size-allocate 15075 gedit(7492): --> 0x0000000002378010[GeditNotebook]::size-allocate 15084 gedit(7492): --> 0x000000000235ac30[GeditTab]::size-allocate 15092 gedit(7492): --> 0x00000000023b79f0[GtkScrolledWindow]::size-allocate 15101 gedit(7492): <-- 0x00000000023b79f0[GtkScrolledWindow]::size-allocate 15107 gedit(7492): <-- 0x000000000235ac30[GeditTab]::size-allocate 15119 gedit(7492): --> 0x00000000023b9df0[GtkHBox]::size-allocate 15128 gedit(7492): <-- 0x00000000023b9df0[GtkHBox]::size-allocate 15134 gedit(7492): <-- 0x0000000002378010[GeditNotebook]::size-allocate 15140 gedit(7492): <-- 0x000000000236f190[GtkVPaned]::size-allocate 15146 gedit(7492): <-- 0x000000000236f090[GtkHPaned]::size-allocate 15153 gedit(7492): <-- 0x00000000021a2950[GtkVBox]::size-allocate 15159 gedit(7492): <-- 0x00000000021c2040[GeditWindow]::size-allocate 15165 gedit(7492): <-- 0x00000000021c2040[GeditWindow]::check-resize
This is really cool and useful stuff.
Unfortunately the current systemtap static marker implementation is a bit inefficient, so I wouldn’t at the moment enable this for a shipping production glib. However, this can and will be fixed (I even proposed some ideas of how to do this in the systemtap bugzilla).
New member in the family
On wednesday 19:13 (CET) my wife gave birth to a boy of 4.1 kg. Both mother and son are well.
So, I’ll be rather busy with non-computing things in the near future. Don’t expect replies or feedback from me.
GObject performance on 64bit arches
I just ran pahole on a Nautilus and got this:
struct _GObject {
GTypeInstance g_type_instance; /* 0 8 */
volatile guint ref_count; /* 8 4 */
/* XXX 4 bytes hole, try to pack */
GData * qdata; /* 16 8 */
/* size: 24, cachelines: 1, members: 3 */
/* sum members: 20, holes: 1, sum holes: 4 */
/* last cacheline: 24 bytes */
}; /* definitions: 138 */
Obviously this is a 64bit machine, so the qdata has to be 64bit aligned, but the ref_counter is only 32 bit. This leaves us with 32 bit of unused space. And at the same time we do all sort of bad-ass slow atomic hacks to extract the low two bits of the qdata pointer.
We could easily use two of these bits instead of the qdata bits on 64bit machines, and avoid lots of unnecessary atomic complex handling of qdata. And we might be able to use these extra bits for other (non-mandatory) performance tricks.
The gospel of git: Interactive Rebase
Today I saw this in my scrollback buffer:
<believer1> so, I changed the patch to not break dbus proto, tested it ... problem is,
that it is followed by the sftp bit ... what is the best practice?
cherry-pick to the different branch apply and amend? or reset apply amend?
<believer1> any special tricks?
<believer2> checkout the commit you wanna edit into a different branch, change it, then
rebase the original branch on top of it
<believer2> i think that's what i always do
<believer1> sounds reasonable
And it was good. These disciples are trying to follow the holy commandments of git, including the important:
Thou shalt keep a clean version history
However, while they are good of heart, doing the best they can, they are not properly enlightened on the book of rebase in the gospel of git. Fortunately such enlightenment is easily come by with some reading and training.
The first rule when it comes to rebases is:
Don’t fear the rebase
Rebasing is history rewriting operation, and most fear of the rebase come from the fear of losing ones history. This is only natural, after all those who lose their history are doomed to rewrite it again. However, once a change is comitted git never loses it (unless you manually gc which I recommend never doing). If you rewrite the history of a branch and the branch now points to this new history, the old commits and history are still stored in your .git directory. All you need to know is the sha1 id of the last commit before your rewrite and you can always access it. For instance, if you’re on a branch that you completely messed up you can get everything back using:
git reset --hard <old-sha1>
This will reset the current branch so that it points to the old history.
If you’re uncertain of how rewriting works and fear something will go wrong, just store away the previous commit which you can easily get using e.g. gitk (or using “git rev-parse HEAD” if you want to be hardcore). And, if you forgot to do this, things are not lost, you can use “git reflog” to find the old history.
So, having overcome our fear of the rebase, how does one actually use interactive rebasing in this case? The background is that you have a bunch of local commits on you branch, each implementing a small independent change (so called microcommits), but before this is applied or merged into a public repo you discover a bug in one of the changes in the middle of the series. It would be nice if such a bug was never visible in the final version history. What to do?
Typically what I do is commit the fix like usual, with a short commit message like “Fixed up the foobar change”. Then you start the interactive rebase using “git rebase -i origin“. This will bring up your chosen editor with a bunch of lines in them, looking something like this:
pick fa1afe1 Implement the foobar function pick 124efd3 Use the foobar function pick cafe123 Fixed up the foobar change
Each line corresponds to one commit on your branch that will be applied during the rebase. You can change the order of the lines to change the history order, or you can delete lines to drop certain commits. What we want to do here is move the fixup to just after the right commit and then change “pick” to “squash” (or just “s” for short) which will merge the two changes into one. So change it to this, save and exit:
pick fa1afe1 Implement the foobar function s cafe123 Fixed up the foobar change pick 124efd3 Use the foobar functionThis will bring up an editor with the commit messages for the first two commits which you edit (often just remove the second one), save and exit. Then the rest of the rebase is done and we end up with a clean history.
The true disciple of git always uses a local branch for each feature he works on, and then when it is done, it is merged into master and then pushed. I must confess that I sometimes sin here, doing minor feature development directly on master, rebasing to clean up and then just pushing that. Sometimes when I do this it turns out that the feature was more complicated than I expected and I didn’t finish it before I had to work on something else. If you end up in this situation its very nice to know that you can then create a branch afterwards by just doing:
git branch <branchname> git reset --hard origin/masterThis (if run on master branch with “origin” being the upstream repo) will create a new branch with the given name that contains your local changes, then it will change the master branch so that it points to what upstream points to, allowing you do do other work and return to the branch later.
gdb over irc
Python scripting in gdb 7.0 kicks total ass. Today I played around with one cool usecase for it: Remote debugging.
I don’t know how many times I have tried to help someone debug over irc, with the person cutting and pasteing gdb commands and results into xchat. Well, no more! Today I hacked up a gdb python script and an xchat python script that lets you export a gdb session over irc:
From gdb:
(gdb) source gdb-server.py gdb irc server, Waiting for connection on /tmp/gdb-socket-500
In xchat:
> /load ~/xchat-gdb.py Loaded xchat-gdb > /join #test > /gdb connect connecting to gdb... connected!
Loggin in as another user on irc:
> /join #test > alex: gdb print "yey" <alex> $6 = "yey" > alex: gdb bt 2 <alex> #0 0x00000035010d50d3 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1) at .. +/sysdeps/unix/sysv/linux/poll.c:87 <alex> #1 0x00007ffff52f96cc in g_main_context_poll (n_fds=<value optimized out>, fds=<value optimized out>, +priority=<value optimized out>, timeout=<value optimized out>, context=<value optimized out>) at gmain.c:2904
Obviously this is kinda unsafe as you can do all sorts of things via gdb. It needs to be able to limit who can control your gdb instance, and you should only give such permissions to people you trust.
Git repo here, have fun with it.
archer gdb macros for glib
The Archer project is working on modernizing gdb. One aspec of this is support for scripting gdb using python. Today I landed some python macros for gdb that makes debugger integration with glib/gobject much nicer.
This is best shown by a “screencast” showing the new features. If you’ve ever debugged a glib program, read the below carefully. If not, it won’t make much sense to you, sorry.
(gdb) # Welcome to the glib macros demo (gdb) # We support pretty printing of lists: (gdb) l toplevel_list 296 static GtkKeyHash *gtk_window_get_key_hash (GtkWindow *window); 297 static void gtk_window_free_key_hash (GtkWindow *window); 298 static void gtk_window_on_composited_changed (GdkScreen *screen, 299 GtkWindow *window); 300 301 static GSList *toplevel_list = NULL; 302 static guint window_signals[LAST_SIGNAL] = { 0 }; 303 static GList *default_icon_list = NULL; 304 static gchar *default_icon_name = NULL; 305 static guint default_icon_serial = 0; (gdb) p toplevel_list $1 = 0x84d7218 = {0x842f468, 0x842f3b8, 0x842f308, 0x842f258, 0x842f1a8, 0x842f0f8, 0x842f048, 0x8370710, 0x8370660, 0x83705b0, 0x8370500, 0x8370450, 0x83703a0, 0x83702f0, 0x8370240, 0x8370190, 0x83700e0, 0x8370030, 0x8349088} (gdb) # And hashtables: (gdb) l g_quark_ht 88 G_LOCK_DEFINE_STATIC (g_dataset_global); 89 static GHashTable *g_dataset_location_ht = NULL; 90 static GDataset *g_dataset_cached = NULL; /* should this be 91 threadspecific? */ 92 G_LOCK_DEFINE_STATIC (g_quark_global); 93 static GHashTable *g_quark_ht = NULL; 94 static gchar **g_quarks = NULL; 95 static GQuark g_quark_seq_id = 0; 96 97 /* --- functions --- */ (gdb) p g_quark_ht $2 = 0x8220a30 = { [0x824db08 "draw-border"] = 0x97, [0x83bdc38 "<Actions>/IconViewActions/Sort by Size"] = 0x962, [0x3970905 "custom-success"] = 0x414, [0x83c0878 "<Actions>/DirViewActions/OpenFolderWindow"] = 0x8df, [0x2a9d6bb "tooltip-text"] = 0x107, [0x82c0648 "audio_%d"] = 0x50c, [0x83af728 "zoom_level_changed"] = 0x7e6, [0x83c1210 "Empty Trash"] = 0x8e6, [0x83584a0 "GdkPixmap"] = 0x76c, [0x83298d0 "soup"] = 0x61f, [0x82e4a58 "gio"] = 0x568, [0x83c2600 "<Actions>/DirViewActions/Create Link"] = 0x8f9, [0x835bca0 "MimeTypes"] = 0x6ed, [0x81c221d "NautilusIconInfo"] = 0x7bb, [0x2b2ff13 "gtk-file-chooser-backend"] = 0x84, [0x82edee8 "audio/x-shorten"] = 0x5a6, [0x82b9e08 "GstAllocTraceFlags"] = 0x498, [0x8307798 "videotestsrc"] = 0x5ee, [0x82cc6c0 "tcp"] = 0x529, [0x81a3dda "NautilusBookmarkList"] = 0x733, [0xc76e26 "dirname"] = 0x350, [0x2b1f004 "gtk-menu-bar-accel"] = 0x66, [0x8267da8 "GstObjectFlags"] = 0x461, [0x82bea88 "http://farsight.sf.net"] = 0x500, [0x835bbf8 "FileSystems"] = 0x6ec, [0x82b8bd0 "GstDebugColorFlags"] = 0x481, [0x81c564d "EggSMClient"] = 0x2b3, [0x83b6528 "band-select-ended"] = 0x84f, [0xc685fd "GFile"] = 0x318, [0x83bcb80 "row-reordered"] = 0x8ac, [0x8320628 "liveadder"] = 0x60f, [0x825c870 "drag_failed"] = 0x171, [0x825d8e0 "gnome_disable_sound_events"] = 0xd6, [0x2b4f420 "grab-focus"] = 0x129, [0x83c4c20 "<Actions>/DirViewActions/Self Format Volume"] = 0x91d, [0x2b3889d "x"] = 0x817, [0x2a92cc0 "y"] = 0x818, [0x2b2a535 "color-hash"] = 0x83, [0x2b3c0e6 "mask"] = 0x76f, [0x8343360 "<Actions>/ShellActions/Zoom Normal"] = 0x711, [0x83cd8b0 "<Actions>/IconViewActions/Tighter Layout"] = 0x956, [0x2b4f28a "gtk-pango-context"] = 0xea, [0x82ec370 "application/x-gzip"] = 0x58f, ---Type <return> to continue, or q <return> to quit---q Quit (gdb) # We also have a cool gforeach command: (gdb) gforeach i in toplevel_list: print ((GtkWindow *)$i) $3 = 0x842f468 [GtkWindow] $4 = 0x842f3b8 [GtkWindow] $5 = 0x842f308 [GtkWindow] $6 = 0x842f258 [GtkWindow] $7 = 0x842f1a8 [GtkWindow] $8 = 0x842f0f8 [GtkWindow] $9 = 0x842f048 [GtkWindow] $10 = 0x8370710 [GtkWindow] $11 = 0x8370660 [GtkWindow] $12 = 0x83705b0 [GtkWindow] $13 = 0x8370500 [GtkWindow] $14 = 0x8370450 [GtkWindow] $15 = 0x83703a0 [GtkWindow] $16 = 0x83702f0 [GtkWindow] $17 = 0x8370240 [GtkWindow] $18 = 0x8370190 [GtkWindow] $19 = 0x83700e0 [GtkWindow] $20 = 0x8370030 [GtkWindow] $21 = 0x8349088 [NautilusSpatialWindow] (gdb) gforeach i in toplevel_list: print ((GtkWindow *)$i)->title $22 = (gchar *) 0x0 $23 = (gchar *) 0x0 $24 = (gchar *) 0x0 $25 = (gchar *) 0x0 $26 = (gchar *) 0x0 $27 = (gchar *) 0x0 $28 = (gchar *) 0x0 $29 = (gchar *) 0x0 $30 = (gchar *) 0x0 $31 = (gchar *) 0x0 $32 = (gchar *) 0x0 $33 = (gchar *) 0x0 $34 = (gchar *) 0x0 $35 = (gchar *) 0x0 $36 = (gchar *) 0x0 $37 = (gchar *) 0x0 $38 = (gchar *) 0x0 $39 = (gchar *) 0x0 $40 = (gchar *) 0x83d0200 "alex" (gdb) # There is also some nice GObject integration features: (gdb) break gtk_widget_size_allocate Breakpoint 1 at 0x29ff265: file gtkwidget.c, line 3821. (gdb) c Continuing. [Thread 0xb71ffb70 (LWP 2599) exited] Breakpoint 1, IA__gtk_widget_size_allocate (widget=0x8349088 [NautilusSpatialWindow], allocation=0xbfffe914) at gtkwidget.c:3821 3821 { (gdb) # Notice the runtime detected type of the instance (gdb) c Continuing. Breakpoint 1, IA__gtk_widget_size_allocate (widget=0x8313d38 [GtkTable], allocation=0xbfffe460) at gtkwidget.c:3821 3821 { (gdb) c Continuing. Breakpoint 1, IA__gtk_widget_size_allocate (widget=0x8365c70 [GtkVBox], allocation=0xbfffe048) at gtkwidget.c:3821 3821 { (gdb) c Continuing. Breakpoint 1, IA__gtk_widget_size_allocate (widget=0x8365d20 [GtkVBox], allocation=0xbfffdc08) at gtkwidget.c:3821 3821 { (gdb) # This is gonna be a long hairy backtrace, right? NO! (gdb) new-backtrace #0 gtk_widget_size_allocate (widget=0x8365d20 [GtkVBox], allocation=0xbfffdc08) at gtkwidget.c:3821 #1 0x027dcb19 in gtk_box_size_allocate (widget=<value optimized out>, allocation=<value optimized out>) at gtkbox.c:500 #2 0x00a180ac in g_cclosure_marshal_VOID__BOXED (closure=0x8249ad0, return_value=0x0, n_param_values=2, param_values= 0x8535e78, invocation_hint=0xbfffddc0, marshal_data=0x27dc850) at gmarshal.c:566 #3 <...> #4 <...> #5 <...> #6 <...> #7 <emit signal size-allocate on instance 0x8365c70 [GtkVBox]> #8 0x029ff3da in gtk_widget_size_allocate (widget=<value optimized out>, allocation=<value optimized out>) at gtkwidget.c:3887 #9 0x0295b625 in gtk_table_size_allocate_pass2 (table=<value optimized out>) at gtktable.c:1610 #10 0x0295b625 in gtk_table_size_allocate (table=<value optimized out>) at gtktable.c:849 #11 0x00a180ac in g_cclosure_marshal_VOID__BOXED (closure=0x8249ad0, return_value=0x0, n_param_values=2, param_values= 0x85956f0, invocation_hint=0xbfffe200, marshal_data=0x295abd0) at gmarshal.c:566 #12 <...> #13 <...> #14 <...> #15 <...> #16 <emit signal size-allocate on instance 0x8313d38 [GtkTable]> #17 0x029ff3da in gtk_widget_size_allocate (widget=<value optimized out>, allocation=<value optimized out>) at gtkwidget.c:3887 #18 0x02a14b39 in gtk_window_size_allocate (widget=<value optimized out>, allocation=<value optimized out>) at gtkwindow.c:4941 #19 0x00a180ac in g_cclosure_marshal_VOID__BOXED (closure=0x8249ad0, return_value=0x0, n_param_values=2, param_values= 0x859bc28, invocation_hint=0xbfffe610, marshal_data=0x2a149f0) at gmarshal.c:566 #20 <...> #21 <...> #22 <...> #23 <...> #24 <emit signal size-allocate on instance 0x8349088 [NautilusSpatialWindow]> #25 0x029ff3da in gtk_widget_size_allocate (widget=<value optimized out>, allocation=<value optimized out>) at gtkwidget.c:3887 #26 0x02a15044 in gtk_window_move_resize (window=<value optimized out>) at gtkwindow.c:6186 #27 0x02a15044 in gtk_window_check_resize (window=<value optimized out>) at gtkwindow.c:5358 #28 0x00a17994 in g_cclosure_marshal_VOID__VOID (closure=0x8262b68, return_value=0x0, n_param_values=1, param_values= 0x8535700, invocation_hint=0xbfffeae0, marshal_data=0x2a14b50) at gmarshal.c:77 #29 <...> #30 <...> #31 <...> #32 <...> #33 <emit signal check-resize on instance 0x8349088 [NautilusSpatialWindow]> #34 0x028176ba in gtk_container_check_resize (container=<value optimized out>) at gtkcontainer.c:1424 #35 0x028179f2 in gtk_container_idle_sizer (data=<value optimized out>) at gtkcontainer.c:1350 #36 0x002c3588 in gdk_threads_dispatch (data=<value optimized out>) at gdk.c:506 #37 0x00929382 in g_idle_dispatch (source=0x8557f20, callback=0x8365d20, user_data=0x83548c0) at gmain.c:4065 #38 0x0092b198 in g_main_dispatch (context=<value optimized out>) at gmain.c:1960 #39 0x0092b198 in g_main_context_dispatch (context=<value optimized out>) at gmain.c:2513 #40 0x0092eac8 in g_main_context_iterate (context=0x8247f90, block=<value optimized out>, dispatch=1, self= 0x821f018) at gmain.c:2591 #41 0x0092ef3f in g_main_loop_run (loop=0x8291758) at gmain.c:2799 #42 0x028b7129 in gtk_main () at gtkmain.c:1205 #43 0x0807e923 in g_themed_icon_append_name () at gthemedicon.c:378 #44 0x0071ab36 in __libc_start_main (main=0x807e2c0 <g_themed_icon_append_name+86084>, argc=1, ubp_av=0xbffff154, init= 0x81a3180, fini=0x81a3170, rtld_fini=0x6efd00 <_dl_fini>, stack_end=0xbffff14c) at libc-start.c:220 #45 0x080692b1 in g_themed_icon_append_name () at gthemedicon.c:378
The above was recorded using a stock rawhide (to be Fedora 12) gdb and glib (including debuginfo packages), with just the python macros added. It also works with the gdb shipping in Fedora 11, but there is some issue there that makes the gforeach macro crash gdb. Additionally, the VTA work that landed in the GCC in Fedora 12 makes what gdb reports much more reliable wrt optimizations, which is very nice for e.g. the backtrace filtering.
GObject performance work
I spent some time last week and this week on fixing some performance issues in gobject. It started out with the patches in bug 557100, which seemed very useful. I cleaned up those patches a bit, wrote a serious performance test and did some additional optimizations.
These changes focus on speeding up creation of “simple” gobject, i.e. things that have no properties or implement any interfaces, etc. They are still important because being able to use gobject gives us lots of advantages like threadsafe refcounting, runtime type introspection, user-data, etc. Sometimes people avoid using gobjects for small things just because they are a bit more expensive than some homebrew struct, which is very sad. With these fixes we can get rid of some of that.
Another thing about gobject that has bothered me for some time is the handling of interfaces. GIO and other modern APIs are starting to use interfaces more and more, so its important that they work well. However, interfaces in gobject have a feature that most people are unaware of, namely that you can add interfaces to a class after the class/type has been initialized. This means that the list of interfaces a class implements must be protected by a lock, and this lock must be taken each time we e.g. check if an object implement an interface or cast to the interface to do a method call on it.
Additionally the interface lookup algorithm used in gobject uses a binary search on the sorted list of interfaces a class implements. Better approaches are possible, like the one used in gcj (described here) which allows constant time (O(1)) interface lookup.
In bug 594525 and 594650 I described these issues and posted patches that fix them.
I added all these patches to the gobject-performance branch in glib git, including the performance test I wrote. The performance improvements are pretty good:
- Construction speed for simple objects more than doubled, while the construction speed for complex object is not much affected (within one percent).
- Interface typechecks go from 52 to 95 million per second in the non-threaded case and from 12 to 95 if g_threads_init() has been called.
- Additionally the contention for typechecks in multiple threads goes to zero as you can see in the tests does by benjamin in bug 594525.
Data about Data
Warning: Long, technical post
One of the few remaining icky areas of the Nautilus codebase is the metadata store. Its got some weird inefficient XML file format, the code is pretty nasty and its the data is not accessible to other apps. Its been on my list of things to replace for quite some time, and yesterday I finally got rid of it.
The new system is actually pretty cool, both in the API to access is how it works internally. So, I’m gonna spend a few bits on explaining how it works.
Lets start with the requirements and then we can see how to fulfil these. We want:
- A generic per-file key-value store with string and string list values. (String lists are required by Nautilus for e.g. emblems)
- All apps should be able to access the store for both writing and reading.
- Access, in particular read access, needs to be very efficient, even when used in typical I/O fashion (lots of small calls intermixed with other file I/O). Getting the metadata for a file should not be significantly more expensive than a stat syscall.
- Removable media should be handled in a “sane” way, even if multiple volumes may be mounted in the same place.
- We don’t require transactional semantics for the database (i.e. no need to guarantee that a returned metadata set is written to stable storage). What we want is something I call “desktop transaction semantics”.
By this I means that in case of a crash, its fine to lose what you changed in the recent history. However, things that were written a long time ago (even if recently overwritten) should not get lost. You either get the “old” value or the “new” value, but you never ever get neither or a broken database. - Homedirs on NFS should work, without risking database corruption if two logins with the same homedir write concurrently. It is fine if doing so may lose some of these writes, as long as the database is not corrupted. (NFS is still used in a lot of places like universities and enterprise corporations.)
Seems like a pretty tall order. How would you do something like that?
Performance
For performance reason its not a good idea to require IPC for reading data, as doing so can block things for a long time (especially when data are contended, compare i.e. with how gconf reads are a performance issue on login). To avoid this we steal an idea from dconf: all reads go through mmaped files.
These are opened once and the file format in them is designed to allow very fast lookups using a minimal amount of page faults. This means that once things are in a steady state lookup is done without any syscalls at all, and is very fast.
Writes
Metadata writes are a handled by a single process that ensures that concurrent writes are serialized when writing to disk.
Clients talk to the metadata daemon via dbus. The daemon is started automatically by dbus when first used, and it may exit when idle.
Desktop Transaction semantics
In order to give any consistancy guarantees for file writes fsync() is normally used. However this is overkill and in some cases a serious system performance problem (see the recent ext3/4 fsync discussion). Even without the ext3 problem a fsync requires a disk spinup and rotation to guarantee some data on disk before we could return a metadata write call, which is quite costly (on the order of several milliseconds at least).
In order to solve this I’ve made the file format for a single database be in two files. One file is the “tree” which contains a static, read only, metadata tree. This file is replaced using the standard atomic replace model (write to temp, fsync, rename over).
However, we rarely change this file, instead all writes go to another file, the “journal”. As the name implies this is a journal oriented format where each new operation gets written at the end of the journal. Each entry has a checksum so that we can validate the journal on read (in case of crash) and the journal is never fsynced.
After a timeout (or when full) the journal is “rotated”, i.e. we create a new “tree” file containing all the info from the journal and a new empty journal. Once something is rotated into the “tree” it is generally safe for long term storage, but this slow operation happens rarely and not when a client is blocking for the result.
NFS homedirs
It turns out that this setup is mostly OK for the NFS homedir case too. All we have to do is put the log file on a non-NFS location like /tmp so that multiple clients won’t scribble over each other. Once a client rotates the journal it will be safely visible by every client in a safe fashion (although some clients may lose recent writes in case of concurrent updates).
There is one detail with atomic replace on NFS that is problematic. Due to the stateless nature of NFS an open file may be removed on the server by another client (the server don’t know you have the file open), which would later cause an error when we read from the file. Fortunately we can workaround this by opening the database file in a specific way[1].
Removable media
The current Nautilus metadata database uses a single tree based on pathnames to store metadata. This becomes quite weird for removable media where the same path may be reused for multiple disks and where one disk can be mounted in different places. Looking at the database it seems like all these files are merged into a single directory, causing various problems.
The new system uses multiple databases. libudev is used to efficiently look up the filesystem UUID and label for as mount and if that is availible use that as the database id, storing paths relative to that mount. We also have a standard database for your homedir (not based on UUID etc, as the homedir often migrates between systems, etc) and a fall-back “root” database for everything not matching the previous databases.
This means that we should seamlessly handle removable media as long as there are useful UUIDs or labels and have a somewhat ok fall-back otherwise.
Integration with platform
All this is pretty much invisible to applications. Thanks to the gio/GVfs split and the extensible gio APIs things are automatically availible to all applications without using any new APIs once a new GVfs is installed. Metadata can be gotten with the normal g_file_query_info() calls by requesting things from the “metadata” namespace. Similar standard calls can be used to set metadata.
Also, the standard gio copy, move and remove operations automatically affect the metadata databases. For instance, if you move a file its metadata will automatically move with it.
Here is an example:
$ touch /tmp/testfile $ gvfs-info -a "metadata::*" /tmp/testfile attributes: $ gvfs-set-attribute /tmp/testfile metadata::some-key "A metadata value" $ gvfs-info -a "metadata::*" /tmp/testfile attributes: metadata::some-key: A metadata value $ gvfs-copy /tmp/testfile /tmp/testfile2 $ gvfs-info -a "metadata::*" /tmp/testfile2 attributes: metadata::some-key: A metadata value
Relation to Tracker
I think I have to mention this since the Tracker team want other developers to use Tracker as a data store for their applications, and I’m instead creating my own database. I’ll try to explain my reasons and how I think these should cooperate.
First of all there are technical reasons why Tracker is not a good fit. It uses sqlite which is not safe on NFS. It uses a database, so each read operation is an IPC call that gets resolved to a database query, causing performance issues. It is not impossible to make database use efficient, but it requires a different approach than how file I/O normally looks. You need to do larger queries that does as much as possible in one operation, whereas we instead inject many small operations between the ordinary i/o calls (after each stat when reading a directory of files, after each file copy, move or remove, etc).
Secondly, I don’t feel good about storing the kind of metadata Nautilus uses in the Tracker database. There are various vague problems here that all interact. I don’t like the mixing of user specified data like custom icons with auto-extracted or generated data. The tracker database is a huge (gigabytes) complex database with information from lots of sources, mostly autogenerated. This risks the data not being backed up. Also, people having problems with tracker are prone to remove the databases and reindexing just to see if that “fixes it”, or due to database format changes on upgrades. Also, the generic database model seems like overkill for the simple stuff we want to store, like icon positions and spatial window geometry.
Additionally, Tracker is a large dependency, and using it for metadata storage would make it a hard dependency for Nautilus to work at all (to e.g. remember the position of the icons on the desktop). Not everyone wants to use tracker at this point. Some people may want to use another indexer, and some may not want to run Tracker for other reasons. For instance, many people report that system performance when using Tracker suffer. I’m sure this is fixable, but at this point its imho not yet mature enought to force upon every Gnome user.
I don’t want to be viewed like any kind of opponent of Tracker though. I think it is an excellent project, and I’m interested in using it, fixing issues it has and helping them work on it for integration with Nautilus and the new metadata store.
Tracker already indexes all kinds of information about files (filename, filesize, mtime, etc) so that you can do queries for these things. Similarly it should extract metadata from the metadata store (the size of this pales in comparison to the text indexes anyways, so no worries). To facilitate this I want to work with the Tracker people to ensure tracker can efficiently index the metadata and get updates when metadata changes for a file.
Where to go from here
While some initial code has landed in git everything is not finished. There are some lose ends in the metadata system itself, plus we need to add code to import the old nautilus metadata store into the new one.
We can also start using metadata in other places now. For instance, the file selector could show emblems and custom icons, etc.
Footnotes
[1] Remove-safe opening a file on NFS:
Link the file to a temporary filename, open the temp file, unlink the tempfile. Now the NFS client on you OS will “magically” rename the tempfile to .nfsXXXXX something and will track this fd to ensure this gets remove when the fd is closed. Other clients removing the original file will not cause the .nfsXXXX link on the server to be removed.