Nautilus: now eats 33% less babies

The work on the nautilus-gio branch continues:

[alex@localhost nautilus-gio]$ find -name "*.c" | xargs grep gnome_vfs_ |  wc -l
531

Where it originally was:

[alex@localhost nautilus]$ find -name "*.[ch]" | xargs grep gnome_vfs_ |  wc -l
803

So, 33% of all gnome vfs calls have been removed.
This doesn’t seem like a lot, but much of the core of nautilus has moved over.

These are the current uses of gnome vfs in nautilus-gio:

      2 gnome_vfs_async_cancel
      1 gnome_vfs_async_close
      1 gnome_vfs_async_load_directory_uri
      1 gnome_vfs_async_open
      2 gnome_vfs_async_read
      2 gnome_vfs_async_set_file_info
      6 gnome_vfs_async_xfer
      1 gnome_vfs_check_same_fs_uris
      2 gnome_vfs_close
      2 gnome_vfs_connect_to_server
      1 gnome_vfs_directory_list_load
      1 gnome_vfs_directory_visit_uri
      2 gnome_vfs_drive_compare
      3 gnome_vfs_drive_eject
      2 gnome_vfs_drive_get_activation_uri
      1 gnome_vfs_drive_get_device_path
      8 gnome_vfs_drive_get_device_type
      2 gnome_vfs_drive_get_display_name
      1 gnome_vfs_drive_get_icon
      2 gnome_vfs_drive_get_mounted_volumes
      8 gnome_vfs_drive_is_mounted
      1 gnome_vfs_drive_is_user_visible
      5 gnome_vfs_drive_mount
      3 gnome_vfs_drive_ref
      3 gnome_vfs_drive_unmount
     17 gnome_vfs_drive_unref
      1 gnome_vfs_error_quark
      5 gnome_vfs_escape_path_string
      1 gnome_vfs_escape_slashes
      6 gnome_vfs_escape_string
      5 gnome_vfs_file_info_list_free
      8 gnome_vfs_file_info_new
      1 gnome_vfs_file_info_to_gio
      9 gnome_vfs_file_info_unref
      2 gnome_vfs_file_type_from_g_file_type
      2 gnome_vfs_file_type_to_g_file_type
      9 gnome_vfs_find_directory
      5 gnome_vfs_get_file_info
      1 gnome_vfs_get_uri_scheme
      1 gnome_vfs_get_volume_free_space
     18 gnome_vfs_get_volume_monitor
      2 gnome_vfs_init
      1 gnome_vfs_make_directory
      1 gnome_vfs_make_directory_for_uri
      5 gnome_vfs_make_uri_from_input
      1 gnome_vfs_make_uri_from_input_with_trailing_ws
      1 gnome_vfs_make_uri_from_shell_arg
      1 gnome_vfs_method_get
      2 gnome_vfs_monitor_add
      2 gnome_vfs_monitor_cancel
      1 gnome_vfs_open
      1 gnome_vfs_read
      1 gnome_vfs_result_to_error
      7 gnome_vfs_result_to_string
      5 gnome_vfs_shutdown
      4 gnome_vfs_unescape_string
      4 gnome_vfs_unescape_string_for_display
      2 gnome_vfs_unlink
      7 gnome_vfs_uri_append_file_name
      3 gnome_vfs_uri_append_string
      1 gnome_vfs_uri_dup
      7 gnome_vfs_uri_equal
      2 gnome_vfs_uri_exists
      4 gnome_vfs_uri_extract_dirname
      4 gnome_vfs_uri_extract_short_name
      2 gnome_vfs_uri_extract_short_path_name
      7 gnome_vfs_uri_get_host_name
      1 gnome_vfs_uri_get_host_port
      9 gnome_vfs_uri_get_parent
      2 gnome_vfs_uri_get_path
      2 gnome_vfs_uri_get_scheme
      3 gnome_vfs_uri_get_user_name
      3 gnome_vfs_uri_has_parent
      1 gnome_vfs_uri_is_local
      6 gnome_vfs_uri_is_parent
      1 gnome_vfs_uri_list_extract_uris
      8 gnome_vfs_uri_list_free
      2 gnome_vfs_uri_make_full_from_relative
     54 gnome_vfs_uri_new
      4 gnome_vfs_uri_ref
     12 gnome_vfs_uris_match
     18 gnome_vfs_uri_to_string
     68 gnome_vfs_uri_unref
      1 gnome_vfs_url_show_with_env
      4 gnome_vfs_volume_compare
      4 gnome_vfs_volume_eject
     12 gnome_vfs_volume_get_activation_uri
      6 gnome_vfs_volume_get_device_type
      7 gnome_vfs_volume_get_display_name
      2 gnome_vfs_volume_get_drive
      1 gnome_vfs_volume_get_filesystem_type
      5 gnome_vfs_volume_get_icon
      2 gnome_vfs_volume_get_volume_type
      1 gnome_vfs_volume_handles_trash
      1 gnome_vfs_volume_is_read_only
      6 gnome_vfs_volume_is_user_visible
      2 gnome_vfs_volume_monitor_get_connected_drives
      1 gnome_vfs_volume_monitor_get_drive_by_id
      7 gnome_vfs_volume_monitor_get_mounted_volumes
      1 gnome_vfs_volume_monitor_get_volume_by_id
      5 gnome_vfs_volume_monitor_get_volume_for_path
      6 gnome_vfs_volume_ref
      3 gnome_vfs_volume_unmount
     29 gnome_vfs_volume_unref
      1 gnome_vfs_xfer_uri

A lot of these are low hanging fruit. For instance, there are 163 calls to the gnome vfs volume monitoring API, which has more or less a direct translation to GIO. Another large part is uri handling which should be translatable to GFile manipulation with some work. So it should be possible to get these numbers a lot lower fast.

However, other parts, like gnome_vfs_xfer() needs significant work. It is only called once, but is a large part of nautilus functionallity, basically doing all sorts of file copies and moves.

So, we’re going places, but we’re far from there yet.

The revolution is coming

I’ve started working on converting Nautilus to use the gio API instead of gnome-vfs. Its nowhere near done, but today I finished converting the basic file information reading and handling to gio.

Nautilus screenshot

Here it is, in all its glory!

Of course, it looks exactly the same as before, but its new and shiny on the inside. 🙂

The code is available in the “nautilus-gio” branch in subversion. At the moment it only works with the standard gio local files backend. If you install gvfs it crashes due to some dbus mainloop integration conflict issues.

Glick 0.2 released

There was two really embarrasing bugs in the 0.1 release. First of all the argument order was switched around in the description of how to create glicks in the README. Secondly, a bug in the fuse filesystem implementation made it hang after 10 files had been opened.

So, a quick release is in order. Get your fresh new code at the glick home page.

Thanks to  Stefan Westerfeld for finding these issues.

Glick 0.1 released

I’m back from my vacation now, and instead of spending days reading through the backlog of emails I decided to polish up glick and make a release so that people can play with it.

One problem that the initial glick version had was that all file lookups were done via the /tmp/glick_root symlink. This symlink being in /tmp and possibly being owned by someone else is a security issue. So, in the new release we instead use “/proc/self/fd/1023” as the absolute prefix for the glick mount. While this looks a bit strange it is much more secure. However, it does make it a bit more complicated to create and test glick bundles, so glick now ships with the “glick-shell” tool that lets you point to a working directory and make /proc/self/fd/1023 point to that.

The new release also contains an easy to use script “glick-mkext2”  that creates minimally sized ext2 images from a directory, in addition to the mkglick script that creates the actual glick.

I’ve also added –icon and –desktop-file switches to mkglick that lets you embed 48×48 png icons and desktop files into the ELF file. These are stored in the “.xdg.icon.48” and “.xdg.desktop” sections and can be easily extracted (using e.g. objdump, or some simple ELF header parsing code). In fact, glick now also ships with a tool “glick-extract” that lets you extract the filesystem, icon and desktop file parts easily.

I’ve talked to some people about the GPL licensing issue discussed in my previous entry, and I’ve come to believe that its ok to distribute glick bundles that contain non-GPL programs even though the glick code is GPL. Distributing such a bundle is really no different from distributing an iso file with both GPL and non-GPL software, which is explicitly ok due to the aggregation section in the GPL:

In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.

Of course, I am not a lawyer, so seek your own legal advice if you are unsure.

Anyway, this version of glick should be secure, easy to use and useful, please play with it and do amazing things. The source for the release and Fedora 7 rpms are availible from the glick webpage.

Experiments with runtime-less app-bundles

I’ve always admired the klik
project. At least the idea behind it, because the implementation was
(by necessity) a bit ugly. There are two main problems, first it uses
loopback mounts (requires root/setuid, has weird limitations, etc) and
secondly it does relocation of packages (so that they can find their
files) by patching the binaries.

Thats why I was so happy when i saw this blog
entry. The klik developers are working on klik2, which is working
to clean up these issues. I took a quick look at it, and they seem to
be using fuse instead of loopback, which is very good. To solve the
relocation problem they seem to be using fuse to implement a full
union filesystem.

I wasn’t completely sold on the union filesystem thing, as I thought
that was a bit overkill. All that really needs to be virtualized is one
directory, accessible via a fixed name for the running
application. Then you can install things in that given prefix, make a bundle of it and it
will run fine.

A few days later when I was lying on the beach I came up with various ways this
could be done. After a while I realized that it would be
possible to use fuse to do a runtime-less bundle system. In other
words, all you needed to install was a standard fuse, and then you can download the
app as one file and just run it. I’m on vacation and shouldn’t even be
touching my computer, but I was so exited about this idea I just had
to see if I could do it. After a day or so of hacking, i was able to make it work.

Introducing glick

glick bundles are standard ELF files that you can run like any other
binary, but they contain all the files, libs and binaries needed for
the application in that one file. glick files embedd a filesystem in the
rodata section of the elf file, and the startup code contains a fuse
filesystem to mount this filesystem, and some magic glue code to make
it all work together.

Here are two example bundles: fancy
greeter, katach.

I’ve tested these on F7, and it seems to work. (fuse-lib and fuse must be
installed, and due to a broken Fedora packaging of fuse the user must be a
member of the fuse group or fusermount needs to be chmod:ed).

Implementation details

On application launch the glick code is started. It then creates a
temporary directory where the filesytem will be mounted. Then it creates a pipe to
talk to a child process and forks. The child process starts up the
fuse filesystem (reading from the automatically mmap:ed rodata section) and mounts that on the temp directory. Then it talks
to the parent by writing to the pipe.

The parent code opens the directory, and uses dup2() to store the file
descriptor to it as fd number 1023. Then it creates (or ensures that
there already exists) a symlink called “/tmp/glick_root” that points
to “/proc/self/fd/1023”. This is the magic part. Since we have the
target directory opened at fd 1023 the /tmp/glick_root symlink will
point to the mounted fuse filesystem for this process and all its children
(that don’t close fd 1023). This means the code in the filesystem can
hardcode /tmp/glick_root to access their files. (Having this in /tmp
is slightly non-ideal. A better place would be in say /var/glick_root,
but then it wouldn’t be a runtime-less system).

Then it reads from the pipe, waiting for the mount to finish. When the
mount is up it it execs /tmp/glick_root/start which is a script or a
symlink that starts up the real application inside the fuse mount.
When the application dies the pipe to the fuse child is automatically
closed. The child notices this, unmounts the mount, removes the
temporary mountpoint and exits.

Licensing issues?

The implementation uses e2fsprogs-libs to implement the fuse
filesystem (based on fuse-ext2 by Jeff Garzik). This library is
unfortunately GPL, which is not ideal for maximal reuse (my code in glick is X-style
BSD). Does embedding an ext2 filesystem inside an elf
file that contains GPL code make the files in the filesystem a derived
work of the GPL code, or is this a “mere aggregation” in the terms of
the GPL? License questions abound… Anyway, perhaps using fuse-cramfs
that the klik people are working on is a better idea anyway.

The code

The code is availible via git here. Play
around with it, experiment, have fun. I did.

Note: The name “glick” is a pun on the kde:ish name of the the klik
project. This is not ment to be a slight or anything like that. I
greatly appreciate the work of the klik project (although I do believe
that their choice of name unfortunately is costing them mind-share
amongs gnome users). Hopefully the klik people will look at my work and steal the appropriate ideas.

Experiments with clutter

A recurring theme at this years Guadec was the need for a toolkit next to Gtk+ for doing some of the next generation fancy effects that Gtk+ isn’t suited to (i.e. things like coverflow). One of the current experimental approaches for this is Clutter. In order to get a feel for clutter and the requirements and usecases for a toolkit supporting fancy effects I started writing a small demo for a fancy version of a gdm greeter. Its basically a fullscreen face-browser in the form of a Rolodex:

screenshot of fancy-greeter

I did all development on a machine without 3D acceleration (way to go ATI! I’ll never buy your hardware again), and it does work with the Mesa software rasterizer, but its really meant to be running on a 3D accelerated machine, and that is where it looks and behaves nice.

If you want to test it out, the code is available via git at http://www.gnome.org/~alexl/git/fancy-greeter.git/ . You need clutter and clutter-cairo from svn too.

As to clutter, It seems very nice. I got a bunch of OpenGL error spew when removing actors errors that I don’t believe is my fault, so it seems to have some bugs, but it was easy for me to hack up this example in a few hours, so the API is easy to use. I’m not completely sold on the animation framework though, it seems a little hard to use. Hopefully with the competition from other projects like pigment we’ll figure out the best API for this eventually.

Not a lot of time for Gnome atm

I’ve been put (temporary) on the OLPC project at work, working on the update system for that. That, plus the fact that all my spare time (and then some!) are being sucked up by the baby means I have very little time to work on Gnome at the moment.

This means most of my work on gvfs is frozen atm, and I don’t have time to follow the gnome-vfs and nautilus lists much. This is just a temporary thing though, and in a month or two i’ll be back hacking on gvfs!

Translated desktop folders in Gnome/Fedora

For a long time Gnome and other desktops have been using an english directory name for the desktop folder (and a few other common directories like music, download and documents). There has been a lot of discussions (read: flamewars) on how to localize these filename. After the last one I’ve finally broken down and done something about this. So, I present to you xdg-user-dirs.

It is a desktop-agnostic (and dependency free) program that is meant to be integrated by distros so that it gets run very early in the login phase. It will then look at the users locale and system-specific default configurations and use these to create a set of folders in the users homedirectory. It also generates (or updates) a user configuration files that points to these directories so that applications can find them.

I’ve also created the gnome-specific module xdg-user-dirs-gtk and a set of gnome patches that integrates these directories into the Gnome desktop. With these in place (availible in the Fedora 7 rawhide repositories now) users automatically get localized folders like Music, Movies, and Documents, plus default bookmarks to these in the gtk file selector. Any changes to these folders (e.g. moves or renames) with nautilus will automatically be updated in the bookmarks and the user directory configuration.

The Gnome integration even detects when you log in with a new locale and asks you if you want to move your directories to the new language.

Its also very easy for the sysadmin to change the default directories created for users (or to disable the whole thing). Just edit a simple config file.