I’m back! (sorta)

September 22nd, 2008 by alexl

After a long parental leave I’m now partially back to work. I’ll be working half-time until the end of the year.

I’ll try to get a sense of the status of the modules I work on, and I’m glad to help answer the list of questions people have built up over the last 6 months. I’m on email, and will hang out on irc around 9:00 to 15:00 CET.

Async I/O made “easy” using JavaScript

September 16th, 2008 by alexl

Writing code that does async I/O is a total pain in the ass. And I’ve written a whole lot of async I/O code, so I should know.

However, if you’re working in a language with full support for continuations it is a lot easier. JavaScript doesn’t support continuations fully, but it has generators, which is like a lite-version of continuations. That is part of the reason why I’m interested in JavaScript scripting.

So, I’ve been experimenting with  how to bind GIO-style async operations in JavaScript so that its easier to write async I/O code. I’ve got something going now that looks pretty ok.

Here is an example that calls an async function “$open” to read three files. It also demonstrates how to do calls from one async generator to another.

do_stuff = new AsyncRunner (function  (filenames) {
    for (let i in filenames) {
        var data = yield $open (filenames[i], 0);
        print ("data for " + filenames[i] + "  = " + data);
    }
    var ret = yield do_stuff2.call("other.txt");
    print ("AsyncRunner call returned: " + ret);
});
do_stuff2 = new AsyncRunner (function (filename) {
    var data = yield $open (filename, 0);
    print ("data for " + filename + " = " + data);
    yield "the end (of do_stuff2)"
});
do_stuff.start(null, ["test.txt", "test2.txt"]);

Its IMHO pretty easy to understand. At every async op we yield which suspends execution and restarts it at that point when we have the operation results. The “$” in “$open” is just to make it different from the sync variant and still short.

This code is far easier than it would look in C, where we would have to split out a new function at each async function call. The loop in do_stuff makes this quite tricky to do.

The code that implements this is here. Its somewhat complicated, using things like currying, closures and generators, so read it carefully.

I’d like some feedback on this. Are there any tricks I’ve missed that could make this nicer? I really wish javascript had a macro facility so that the yield keyword could be put inside the async calls and we could avoid the boilderplate around the AsyncRunner creation. I would also like to make the AsyncRunner objects callable like “do_stuff2()”, but that doesn’t seem possible with standard javascript (although it is the AsyncRunner class is implemented natively via the spidermonkey APIs…)

Embeddable languages, an implementation

September 9th, 2008 by alexl

I read Havocs post on embeddable languages with great joy. I very much agree that this is a good model for writing a larger application. I.E. you write the core of your application in C/C++ and then do the higher levels in an embedded dynamic scripting language. All the reasons Havoc lists are important, especially for the Gnome platform (a swarm-of-processes model where we want to avoid duplicating the platform in each language used).

There are a lot of dynamic scripting languages around, but only a few of them really qualify as embeddable languages in the form Havoc proposes. I would say the list is basically just: lua, JavaScript and some of the lisp dialects.

I’m far from a JavaScript lover, but it seems pretty obvious to me that from these alternatives it is the best choice. For a variety of reasons:

  • Its a modern dynamic language
    Its dynamically typed, it has precise gargabe collection, lambda functions, generators, array comprehension, etc
  • Its object oriented (sorta)
    We want to wrap the GObject type system which is object oriented, so this is important. The JavaScript prototype-based model is a bit weird, but its simple and it works.
  • It has no “platform” of its own
    If we inject the Gnome platform (Gtk+, Gio, etc) into JavaScript it doesn’t have to fight with any “native” versions of the same functionallity, and we won’t be duplicating anything causing bloat and conversions. (JS does have a Date object, but that is basically all.)
  • Lots of people know it, and if not there are great docs
    Chances of finding someone who knows JS is far higher than finding someone who knows e.g. lua or scheme, and having a large set of potential developers is important for a free software project.
  • Lots of activity around the language and implementations
    The language continually evolves, and there is a standards body trying to shephard this. There is also a multitude of competetive implementations, each trying to be best. This leads to things like the huge performance increases with TraceMoney and Google v8. Such performance work and focus is unlikely to happen in smaller and less used languages.

To experiment with this I started working on GScript (gscript module in gnome svn). It is two things. First of all its an API for easily embedding JavaScript (using SpiderMonkey atm) in a Gtk+ application. Secondly its a binding of GObject and GObject-based libraries to JavaScript.

Here is a quick example:

GScriptEngine *engine;
GScriptValue *res;
engine = g_script_engine_new ();
res = g_script_engine_evaluate_script (engine, "5+5");
g_print ("script result: %s", g_script_value_to_string (res));
g_object_unref (res);

If you want to expose a gobject (such as a GtkLabel) as a global property in JS you can do:

GtkLabel *label = gtk_label_new ("Test");
GScriptValue *val = g_script_value_new_from_gobject (G_OBJECT (label));
g_script_value_set_property (g_script_engine_get_global (engine), "the_label", val);

Then you can access the label from  javascript by the name “the_label”. You can read and set the object properties, connect and emit the signals and call all the methods that are availible via introspection.

You can also easily call from javascript into native code.

static GScriptValue *native_function (int n_args, GScriptValue **args);
GScriptValue *f = g_script_value_new_from_function (native_function, num_args);
g_script_value_set_property (g_script_engine_get_global (engine), "function", f);

The JavaScript wrappers are fully automatic, and lazily binds objects/classes as  they are used in JS. The object properties and signal information are extracted from the GType machinery. Method calls are done using the new GObject-Introspection system.

More work is clearly needed on the details of the JS bindings, but this is already a usable piece of code. I’m very interested in feedback about interest in something like this, and discussions about how the JS bindings should look.

Making backtraces readable

February 7th, 2008 by alexl

Reading gdb backtraces from Gtk+ applications can be a pain, as the signal emissions tend make them very long and hard to read. Today i wrote a small application that compresses signal emissions and some other common gnome stuff in gdb backtraces.

For example, take this 147 frames long backtrace i’m currently working on. Its pretty much a pain to read.

With my app it turns into this instead, which is much nicer. Here are some examples:

#7 ... segfault caught by bug-buddy
#8 <signal handler called> ()
#9 __kernel_vsyscall ()

#36 gtk_scrolled_window_destroy (object=0x82bc538) at gtkscrolledwindow.c:799
     ...
#43 gtk_object_dispose (gobject=0x82bc538) at gtkobject.c:418
#44 gtk_widget_dispose (object=0x82bc538) at gtkwidget.c:7851
#45 fm_tree_view_dispose (object=0x82bc538) at fm-tree-view.c:1457

#115 gtk_object_destroy (object=0x82d0200) at gtkobject.c:403
#120 ... emitting signal 219 on instance 0x5954
#121 _gtk_action_emit_activate (action=0x82c8e60) at gtkaction.c:872

#130 gtk_window_key_press_event (widget=0x82d0200, event=0x83716e8) at gtkwindow.c:4961
#140 ... dispatching GdkEvent 0x83716e8 to widget 0x82d0200
#141 g_main_context_dispatch (context=0x819fca0) at gmain.c:2064

Any chance something like this could be integrated with bugzilla?

Nautilus gio-branch merged – be careful

November 30th, 2007 by alexl

Today I got most of file moving working in nautilus-gio, so that means that most functionality is there, even if its quite raw and not very tested. Its time to get more people involved in testing and working on the new codebase.

So, I just merged the nautilus gio-branch to HEAD.

WARNING: Its still far from done, and it might eat your files. Don’t use in anger! If it breaks you get to keep both pieces.

There is a list of things left todo on the wiki. But even after that is finished there is bound to be a bunch of stuf to be fixed.

So, use with caution, and give feedback.

UPDATE:  Please file bugs against the GIO component of nautilus in bugzilla, so we can track this

gio-standalone merged into glib

November 26th, 2007 by alexl

I just commited the gio code into glib:

185 files changed, 52768 insertions(+), 8 deletions(-)

This is a pretty large piece of work. Its still got some areas that need work, and the docs are far from ready, however the API is imho pretty powerful and easy to understand. If your app uses file I/O in any way, please take a look at the GIO APIs and give feedback.

File operations in nautilus-gio and adventures in the land of PolicyKit

November 23rd, 2007 by alexl

This week I’ve been working on adding file operations to the gio-branch of nautilus. Today I finished enough to hook up a mostly finished copy operation into the UI. This is completely new code, since gio doesn’t have the gnome_vfs_xfer API that implemented complicated file operations in gnome-vfs.

So, while this is doing the same thing as the old code its structured in a completely different way which is much nicer to work with. And along the way I made some other changes that was easy to do with the new improved design. Check out this screencast of the new features:

nautilus-gio-copy.png

Next week I will continue working on the rest of the file operations. It will be a lot easier now that the general structure and a lot of common code is written.

Also, next week I will be starting to merge libgio from the gio-standalone module into the glib module (as a separate libgio-2.0.so). The gio code has all features required by nautilus (as we can see from the almost finished port), and has been pretty API and ABI stable for a while. So, this is a good time to merge it and get more people looking at the code and using it in their applications.

PolicyKit vs Nautilus

But all that is not whats got me most excited right now. Instead its this idea I got about using PolicyKit and gio in order to implement system administrator features in Nautilus. Basically, you’d do a file operation on some system file, which gives you an error dialog saying you don’t have permissions to do this. But the error dialog has a button that lets you authenticate (with e.g. the root password) and continue the operation with root rights.

This is quite simple to implement in nautilus-gio, because all I/O operations go through the GFile interface. You just create an implementation of GFile that proxies all operations to a setuid helper program (which uses PolicyKit to authenticate). With this done, all the Nautilus code will work unmodified with this new backend, all you have to do is make sure it uses this new GFile implementation.

This is much better than running your entire Nautilus process (or even desktop) as root, for multiple reasons. First of all your Nautilus application will integrate nicely with your desktop (using your theme/prefs/etc instead of the root ones). Secondly a lot less code is running as root, which means that the risk for a security breach or accidents as root are much smaller. Third, you can configure this to use sudo-style authentication which means you don’t need to give out the root password (or even have one) .

Still, this work will have to wait, as the main priority right now is getting the gio-branch into a useful shape so that it can be merged into svn HEAD. I think this will happen soon. Maybe next week even.

Last gnome-vfs symbol gone!

November 2nd, 2007 by alexl

The nautilus gio-branch today reached a major milestone. There is now zero references to gnome-vfs symbols in the nautilus binary. This was accomplished by disabling parts of the file operations code in nautilus, so the resulting nautilus can’t actually do some operations. However, all the file browsing and launching code is working.

At this point we’re getting close to a state where we can get people to start testing the new codebase, and thinking about how to integrate it into Gnome 2.22 and glib. To get this going, and to get more people involved with gio I’ve started a wiki page listing some of the things that remain to do in the various levels of the gio stack.

Consider this a call for action. If you’re interested in this, please take a look at the gio APIs, try it out, and if you’re even more adventurous, pick something from the TODO list and start working on it.

gnome-vfs is going down, down, down

October 26th, 2007 by alexl

Work on the Nautilus gio-branch continues. We’re now down to 203 gnome-vfs calls in nautilus (803 initially) and zero in eel (92 originally). Now, 203 calls still sounds like a lot, but lets take a look at what uses gnome-vfs:

  1 libnautilus-private/nautilus-file-operations-progress.c
  3 libnautilus-private/nautilus-file.c
  4 libnautilus-private/nautilus-program-choosing.c
126 libnautilus-private/nautilus-file-operations.c
  3 libnautilus-private/nautilus-vfs-utils.c
  5 libnautilus-private/nautilus-file-utilities.c
  9 libnautilus-private/nautilus-dnd.c
  7 src/file-manager/fm-tree-view.c
  4 src/file-manager/fm-icon-view.c
  1 src/file-manager/fm-directory-view.c
  1 src/file-manager/fm-error-reporting.c
  8 src/file-manager/fm-directory-view-old.c
  5 src/nautilus-main.c
  1 src/nautilus-window-manage-views.c
 23 src/nautilus-connect-server-dialog.c

Almost all of it is in nautilus-file-operations.c, which is the implementation of things like copying/moving files, and displaying progress and other dialogs for that. This is certainly very important functionality, and it will be a bunch of work converting it, especially since it heavily uses the gnome_vfs_xfer API, which (thankfully) doesn’t exist in gio. But apart from that almost all gnome-vfs use in Nautilus is gone

I’ve also cleaned up the code in several places, removing crufty, old and unused code, in places replacing it with more modern Gtk+/glib functionality. Eel especially has dropped a lot of code. Lots of creds go to Paolo Borelli who has done a bunch of work on this.

On a slightly more end-user interesting tangent, today I updated the fuse code that hpj wrote (it had stopped working due to some internal changes) and integrated it with the gvfs backend for libgio. This means g_file_get_path() will now return a local fuse path even for non-local files. In practice this mean applications that do not access files via gio can still load and save files on e.g. remote shares like smb, ftp, sftp, etc.

I made a screencast to show this off. Check it out!

You can pick any new feature you want, as long as it is this one

October 16th, 2007 by alexl

This last week I’ve been working on the icon and thumbnail handling on the nautilus gio-branch. gnome-vfs doesn’t really handle icons at all in the API, so all the code related to selecting and rendering icons for files was done entirely in nautilus. The way this was done was also a bit weird compared to how other things worked.

Nautilus has this abstraction called NautilusFile that contains all the information we know about the file. You can request for it to load specific information, and to get notified when it changes. However, icons were not handled by this. Instead there is a class called NautilusIconFactory that lets you map from file to icon (and incidentally handle things like icon caching, loading thumbnails, etc). Sometimes the icons can change though, like when the icon theme is changed, so all consumers had to also watch the icons_changes signal on the icon factory (in addition to the changed signal on the NautilusFile).

gio adds support for icons (and to some extents thumbnails) directly in the vfs. In the normal case we decide what icon to use for a file in the same way (based on the mimetype). However, this happens in the gio backend, so if the backend has special needs for icons it can do whatever it wants.

This is great stuff and will allow backends to be more expressive, and applications need less code to handle icons. It does however mean that the code in Nautilus doesn’t really match the new APIs. So, since I had to completely restructure the code anyway, its with great pleasure I can announce that since yesterday NautilusIconFactory has been removed from the gio-branch, and the new icon and thumbnailing code is integrated with the NautilusFile object in a much saner way.

Most of this work is just changes needed to work with gio rather than new features. I did however add one new feature. If you resize icons so that they are larger that the thumbnail generated for them (i.e. 128 pixels) they look very blurry. I’ve seen a lot of people with such scaled up pictures of pets or loved ones on the desktop, and it looks quite ugly. So, I made the thumbnail loader try to use the file itself as thumbnail if the icon is scaled above 128 pixels:


Wax On!


Wax Off!

None of this work significantly brings down the number of gnome-vfs calls in nautilus mentioned in my last blog. We’re now down from 803 to 510 compared to 531 in last entry. I think I will need to convert the GnomeVFSVolumeManager calls to gio now, as that will bite of a large chunk of this.