Category Archives: Hacking

Dave Neary on forking

As someone who witnessed Shotwell get forked over a dispute of two text labels, I have to chuckle and nod my head at Dave Neary’s apt analogy:

A fork is like a schoolyard football game. Tommy brings a ball to school, and gets to be captain of a team every time, picks the best players for himself, because if he doesn’t, he threatens to take his ball away. After a few weeks of this, the other players get annoyed, and everyone chips in to buy a ball for the group. At this point, Tommy has lost all power – he still has his ball, but he’s not team captain any more. He gets left standing against the wall till the very end to teach him a lesson in humility. So he sits out the games for a while in protest, starts a competing game at the other end of the yard with his ball and his rules. All the best players go to the other game, though, and some of the kids make fun of “Tommy’s” game – invitations to give up his game & join the new one seem more like taunts than honest efforts to include him. After a while, he realises that playing football with others is more fun, and more important, than playing with his football.

My only comment here is that Tommy in this situation represents a master trunk and the others are the fork.  There’s an additional football analogy here, that of Tommy wanting to be captain and bringing his own ball to the schoolyard and trying to get the others to play his game.  If they do, we admire the power of the open-source ecosphere.  If they don’t, it’s another branch that’s shriveled and died.

In this sense, open-source forking has made it into Urban Dictionary.

Those “realize” & “map” widget signals

I’ve always been a little fuzzy on the relationship between these important Gtk.Widget signals: “map”, “map-event”, “realize”, “show”, and “expose-event”.  I was working on a problem with Shotwell today and realized (no pun) that I needed to intercept at least one of these to solve a UI initialization problem.  Okay — which one?

They’re all related, of course.  Over time I’ve gathered some notion of their order (“realize” before “map” followed at some point by “expose-event”), but what is “map-event” versus “map”?  And why a “show” signal when all these other signals seem to be saying, in their own way, the same thing?

For the nth time I dug through DevHelp (no help) and the Xlib Programming Manual (which is written in some variant of Middle English — I recognize the language, fuzzily) looking for help.  I finally discovered this clue by none other than Havoc Pennington:

Realize means to create the GDK resources for a widget. i.e. to instantiate the widget on the display. This is more useful once we have multiple display support in GTK.

Map means to actually pop the widget’s window onscreen. It requires the widget to be realized, since the window is created in realize.

Show means the widget should be mapped when its toplevel is mapped, or in the case of a toplevel, should be mapped immediately.

Mapping is asynchronous though; that is, gtk_widget_map() and the map signal are emitted when the map is requested. When the map actually occurs you get a map_event (distinct from plain map). But you are not allowed to draw on widgets until you get the first expose, map_event is insufficient.

I’m not saying that clears everything up, but it gave me a starting point:

realize is what takes a data structure in memory (the Widget) and allocates GDK resources (i.e. a window) for it.  I’ve done a lot of network programming in my past, so I imagine this step as analogous to initializing a Session object.  The Session object itself is (usually) cheap to instantiate, but can’t do much until a network connection (i.e. a socket) is created — sockets being a system resource much scarcer than heap memory.

map, as Havoc says, “pops” the window onto the display.  But actually, it only requests the window be displayed.  I compare this to an asynchronous connect() call.  The request is initiated, but when the call returns, the circuit is not complete.

map-event is a GDK event.  This is called when the window is now on-screen, i.e. the connection is complete.  It’s like a callback.

expose-event is called when the window may be drawn upon, and is called thereafter whenever a portion of the window has gone dirty and must be redrawn.  In network parlance, this is like a signal some stacks offer to indicate when a connection is able to send().  (This is the roughest analogy of them all.)

This leaves unrealize, unmap, and unmap-event, which look like obvious complements — signals fired during tear-down to indicate the stages of destroying the underlying window object.

I built a test application in Vala to verify some of this as well as determine the ordering of these signals for a Gtk.Window.  (They might be in a different order for windowed widgets; I didn’t investigate.)  This is the ordering of the events after I called show_all() on the Gtk.Window and then closed the window by clicking on the dismiss button in the chrome:

  1. show
  2. realize
  3. map
  4. expose-event
  5. map-event
  6. unmap
  7. unrealize

Notice that “unmap-event” was never called; this may have to do with connecting the “destroy” signal to Gtk.main_quit().  The fact that the application can exit without it being properly called is enough for me to know not to rely on it during tear-down.

Also notice that “expose-event” is called in between the “map” call and the “map-event” callback.  This too is not what I expected.

And what about “show”?  I understand what the virtual method is doing, but why make it a signal?  Havoc explains in a follow-up:

It’s one of those weird historical things from pre-1.0.

Okay.  I can accept that.

(Postscript: I know that the “expose-event” signal is deprecated in GNOME 3.0 in favor of “draw”, but I still think the above information is useful.)

Vala and the comma operator

One of the more interesting parts of working on a Vala project is occasionally having to read through the C source the Vala compiler generates.  Sometimes it’s to investigate a bug (maybe your own, maybe the compiler’s) and sometimes it’s just out of curiosity.

Personally, I think the C code Vala produces is pretty darn good as far as machine-generated code goes.  (And I’ve seen some horrible machine-generated code, not including obfuscators and their ilk.  Which reminds me, have I ever told you about when I met Phil Katz?  Another time.)  Certainly I feel I could produce much prettier code by hand, but that’s the point: I would have to do it by hand.  A lazy programmer is a good programmer.

Take a look at this typical example of Vala’s generated C code:

gboolean dimensions_has_area (Dimensions *self) {
    gboolean result;
     gboolean _tmp0_ = FALSE;
     if ((*self).width > 0) {
         _tmp0_ = (*self).height > 0;
     } else {
         _tmp0_ = FALSE;
     }
     result = _tmp0_;
     return result;
}

If an interview candidate at Yorba put this up on the whiteboard, I imagine the first thing we’d ask would be, “Ah … anything you think you could do to clean up this function?”

This isn’t to knock Vala or the compiler or the wonderful work Jürg and Team Vala have done.  And yes, I know, gcc will optimize this down to machine code that would be pretty much like the machine code it would produce if I hand-coded this.  And, of course, unless this is in hot loop (which it’s not, last time I checked), none of this matters anyway.

In short, I lose zero sleep over this.

But consider this line, which is also not atypical:

scaled = (dimensions_floor ((_tmp2_ = (dimensions_init (&_tmp0_, (gint) round (scaled_width), (gint) round (scaled_height)), _tmp0_), &_tmp2_), (_tmp3_ = (dimensions_init (&_tmp1_, 1, 1), _tmp1_), &_tmp3_), &_tmp4_), _tmp4_);

Ah … okay, hold on.  Don’t run off.  It’s not as bad as you think.  It’s just that this makes much more sense to the C compiler than it does to you.

My advice to anyone who dives into Vala’s generated code is to study up on the comma operator.  I suspect most programmers (maybe 98% of them!) have never used the comma operator outside a for loop, and even then, rarely.  Most of them would say that the comma operator is only used to perform multiple statements (actually, “operands” in the parlance) on a single line of code.  That is, it’s a handy way when writing a for loop to squeeze in multiple statements inside each of its three parameters.

Which is true, save for one thing: With the comma operator, the statement as a whole evaluates to the final operand.  Thus, the initializers in this for loop evaluate to 42:

for (i = 0, j = 42; i < j; i++)

The initializer isn’t being assigned to anything, so we don’t notice, so we never learn.

Looking back over that C code up there, mentally weed out the _tmp0_, _tmp1_, etc. and see it for what it is: Initializing two dimensions structs (one goes into _tmp2_, the other into _tmp3_), which are passed to dimensions_floor as parameters, which puts its result into _tmp4_ (whose address is passed to dimensions_floor as its third parameter).  The final operand of the comma operator is, in fact, _tmp4_, which then is returned to scaled.  The comma operator makes all this happen.

Here’s that line in Vala, by the way:

Dimensions scaled = Dimensions((int) Math.round(scaled_width), (int) Math.round(scaled_height)).floor();

Bruno Girin on importing photos from other apps to Shotwell

Bruno Girin, author of the F-Spot import subsystem now shipping in Shotwell 0.7, has written a fantastic blog post on his architecture and how to write import code for other photo applications.  He mentions importing from Picasa, which we certainly are interested in doing at some point.  Bruno has done quite a bit of database work and his thoughts and approach are worth a read for those insights alone.

Contributing to Shotwell

Jorge Castro asked for our thoughts on a recent (and highly active) contributor, Bruno Girin.  Bruno provided a mountain-load of energy developing the F-Spot migration code for Shotwell 0.7.  Here’s Jorge’s report as well as Bruno’s story about how he came to work on Shotwell in such an in-depth manner.

We value contributions here at Yorba, and we do our best to evaluate every patch that comes in.  Sometimes the timing might be off — especially if we’re rushing to clean up the final details of a release — but even then we might slip in oh-so-good stuff, such as tag autocompletion that Marcel Stimberg valiantly got in under the wire in time for 0.7.

If there’s something in Shotwell you’d really like to see, if you have an itch to scratch, please consider contributing!  Things you can do to get started (in no particular order):

1. Check out our ticket database for bugs and feature wishes.

2. If you’re not familiar with Vala, acquaint yourself with it — Shotwell is 99.9% Vala.  Fortunately Vala is very similar to Java and C#, so if you’re familiar with either or both, you have a head-start.  Some good Vala resources include its documentation and tutorials, the Valadoc for the various bindings that come with it (here’s the latest Valadoc for Gee, the collection class we use extensively in Shotwell), and my article on The GNOME Journal about why Yorba chose Vala in the first place.

3. Download the source from our Subversion repository: svn co svn://svn.yorba.org/shotwell/trunk

4. Read over Shotwell’s architecture overview and Yorba’s coding guidelines.

Hopefully this gives you a good start on hacking Shotwell.  We look forward to your patch!

Synchronization in Vala

When I first began working with threads in Vala, I was happy to see its lock keyword.  Compare that response to when I wrote my first Java program (wa-aa-ay back in 1996).  I was a bit put off by the synchronized keyword — I still had in me that C-inspired purity that there should be a strict separation between language and library.  I was blind to the idea that critical sections and semaphores should be built into a “serious” language.  I’m sure I muttered something about syntactic sugar and sighed and gave Java threads a try … and loved them.  Critical sections and block-scoped programming really do go hand-in-hand.  It was a joy to know that I could never leave a critical section locked when I exited a function.  That and Java’s intelligent exception handling really sold me on the language.

But when I began coding threads in Vala, I noticed some limitations to the lock keyword.  Firstly (and oddly), the compiler would only allow me to lock member variables of the current object, but not the object itself.  This is perfectly legal in Java:

public void method() {
    synchronized (this) {
        // protected code
    }
}

This is so useful, Java has syntax for locking this throughout an entire method:

public synchronized void method() {
    // protected code
}

So why can’t Vala do something similar?  Why can’t I lock(this) — especially since, if the this variable is a member of another object, that object can lock it!

And I noticed something even odder with lock.  When you synchronize on an object in Java, you’re essentially using an invisible lock() method on the object.  (This is the “intrinsic lock” or “monitor lock” in Java-speak.)  Beneath the covers it’s something like this:

object.lock();
try {
    // protected code
} finally {
    object.unlock();
}

You can see why this will compile in Java but produce its ill-named NullPointerException:

Object object = null;
synchronized (object) {
    // ka-boom
}

However, this code works just fine in Vala:

class Foo {
    Object object = null;
    public void go() {
        lock (object) {
              // smooth sailing
        }
    }
}

What’s the story?  If object is null, how can it be locked?

It turns out that Team Vala have come up with an interesting paradigm for locking variables that (a) works around certain limitations of GObject, and (b) actually cleans up some of my issues with Java-style critical sections.

The key to understanding the lock operator in Vala is to understand that you’re not locking the variable itself.  You’re locking the memory used to store that variable. That’s why Vala can only lock member variables of a class.  The only other storage to lock is the stack, which is rarely (if ever) useful (and probably indicates a design flaw), or global memory, which isn’t such a hot idea either.  (Locking a static member of a class is allowed.)  Jürg discussed this on the Vala mailing list.

Like Java, lock is calling a “magic” reentrant mutex that’s associated with the member variable.  This mutex cannot be reliably passed around with the variable (which would be necessary for code in another class to lock it).  And GObject doesn’t provide a built-in object-wide mutex that Vala can use.  Vala could magically generate one in your own classes, but lock wouldn’t work on any non-Vala generated GObject — including GTK, GLib, and so on.  Hence, this rule is also a nod to the limitations of the libraries Vala is building upon.  To put it another way, lock always works because it can only lock Vala-produced objects.  Ta-da.

And you can lock native types, not just objects.  Locking an int counter is a snap in Vala without resorting to Java’s coarse-grained synchronized(this) or a dummy Object.

Another advantage to Vala’s approach is that it solves one of my gripes of Java synchronization.  Going back to the analogy I mentioned earlier, every Object in Java has an invisible lock() and unlock() method — but those methods are public.  Anybody can lock an object and hold that lock during their six-deep nested loops or blocking calls, even if that lock is relied upon elsewhere in the system (by, say, a thread that very object created and maintains).

Vala doesn’t have this problem.  A member variable is only lockable by methods of its class.  Problem solved.

And go back to the other example, where Java will throw a NullPointerException when synchronizing on a null object.  Because Vala is locking the storage of the reference, and not the reference itself, you can do this reliably and easily:

// atomic singleton creation
lock (object) {
  if (object == null)
    object = new Object();
}
// atomic replacement of multiple fields
lock (object) {
  object = new_object;
  object_modifier = new_object_modifier;
}

You can do the second one in Java, but it’s probably not what you want — most Java programmers will synchronize on this, which may be too coarse.  The only alternative is to generate a dummy Object to lock upon, which is kind of ugly.  Nice and clean in Valaworld.