I just commited the gio code into glib:
185 files changed, 52768 insertions(+), 8 deletions(-)
This is a pretty large piece of work. Its still got some areas that need work, and the docs are far from ready, however the API is imho pretty powerful and easy to understand. If your app uses file I/O in any way, please take a look at the GIO APIs and give feedback.
This week I’ve been working on adding file operations to the gio-branch of nautilus. Today I finished enough to hook up a mostly finished copy operation into the UI. This is completely new code, since gio doesn’t have the gnome_vfs_xfer API that implemented complicated file operations in gnome-vfs.
So, while this is doing the same thing as the old code its structured in a completely different way which is much nicer to work with. And along the way I made some other changes that was easy to do with the new improved design. Check out this screencast of the new features:
Next week I will continue working on the rest of the file operations. It will be a lot easier now that the general structure and a lot of common code is written.
Also, next week I will be starting to merge libgio from the gio-standalone module into the glib module (as a separate libgio-2.0.so). The gio code has all features required by nautilus (as we can see from the almost finished port), and has been pretty API and ABI stable for a while. So, this is a good time to merge it and get more people looking at the code and using it in their applications.
But all that is not whats got me most excited right now. Instead its this idea I got about using PolicyKit and gio in order to implement system administrator features in Nautilus. Basically, you’d do a file operation on some system file, which gives you an error dialog saying you don’t have permissions to do this. But the error dialog has a button that lets you authenticate (with e.g. the root password) and continue the operation with root rights.
This is quite simple to implement in nautilus-gio, because all I/O operations go through the GFile interface. You just create an implementation of GFile that proxies all operations to a setuid helper program (which uses PolicyKit to authenticate). With this done, all the Nautilus code will work unmodified with this new backend, all you have to do is make sure it uses this new GFile implementation.
This is much better than running your entire Nautilus process (or even desktop) as root, for multiple reasons. First of all your Nautilus application will integrate nicely with your desktop (using your theme/prefs/etc instead of the root ones). Secondly a lot less code is running as root, which means that the risk for a security breach or accidents as root are much smaller. Third, you can configure this to use sudo-style authentication which means you don’t need to give out the root password (or even have one) .
Still, this work will have to wait, as the main priority right now is getting the gio-branch into a useful shape so that it can be merged into svn HEAD. I think this will happen soon. Maybe next week even.
The nautilus gio-branch today reached a major milestone. There is now zero references to gnome-vfs symbols in the nautilus binary. This was accomplished by disabling parts of the file operations code in nautilus, so the resulting nautilus can’t actually do some operations. However, all the file browsing and launching code is working.
At this point we’re getting close to a state where we can get people to start testing the new codebase, and thinking about how to integrate it into Gnome 2.22 and glib. To get this going, and to get more people involved with gio I’ve started a wiki page listing some of the things that remain to do in the various levels of the gio stack.
Consider this a call for action. If you’re interested in this, please take a look at the gio APIs, try it out, and if you’re even more adventurous, pick something from the TODO list and start working on it.
Work on the Nautilus gio-branch continues. We’re now down to 203 gnome-vfs calls in nautilus (803 initially) and zero in eel (92 originally). Now, 203 calls still sounds like a lot, but lets take a look at what uses gnome-vfs:
1 libnautilus-private/nautilus-file-operations-progress.c
3 libnautilus-private/nautilus-file.c
4 libnautilus-private/nautilus-program-choosing.c
126 libnautilus-private/nautilus-file-operations.c
3 libnautilus-private/nautilus-vfs-utils.c
5 libnautilus-private/nautilus-file-utilities.c
9 libnautilus-private/nautilus-dnd.c
7 src/file-manager/fm-tree-view.c
4 src/file-manager/fm-icon-view.c
1 src/file-manager/fm-directory-view.c
1 src/file-manager/fm-error-reporting.c
8 src/file-manager/fm-directory-view-old.c
5 src/nautilus-main.c
1 src/nautilus-window-manage-views.c
23 src/nautilus-connect-server-dialog.c
Almost all of it is in nautilus-file-operations.c, which is the implementation of things like copying/moving files, and displaying progress and other dialogs for that. This is certainly very important functionality, and it will be a bunch of work converting it, especially since it heavily uses the gnome_vfs_xfer API, which (thankfully) doesn’t exist in gio. But apart from that almost all gnome-vfs use in Nautilus is gone
I’ve also cleaned up the code in several places, removing crufty, old and unused code, in places replacing it with more modern Gtk+/glib functionality. Eel especially has dropped a lot of code. Lots of creds go to Paolo Borelli who has done a bunch of work on this.
On a slightly more end-user interesting tangent, today I updated the fuse code that hpj wrote (it had stopped working due to some internal changes) and integrated it with the gvfs backend for libgio. This means g_file_get_path() will now return a local fuse path even for non-local files. In practice this mean applications that do not access files via gio can still load and save files on e.g. remote shares like smb, ftp, sftp, etc.
I made a screencast to show this off. Check it out!
This last week I’ve been working on the icon and thumbnail handling on the nautilus gio-branch. gnome-vfs doesn’t really handle icons at all in the API, so all the code related to selecting and rendering icons for files was done entirely in nautilus. The way this was done was also a bit weird compared to how other things worked.
Nautilus has this abstraction called NautilusFile that contains all the information we know about the file. You can request for it to load specific information, and to get notified when it changes. However, icons were not handled by this. Instead there is a class called NautilusIconFactory that lets you map from file to icon (and incidentally handle things like icon caching, loading thumbnails, etc). Sometimes the icons can change though, like when the icon theme is changed, so all consumers had to also watch the icons_changes signal on the icon factory (in addition to the changed signal on the NautilusFile).
gio adds support for icons (and to some extents thumbnails) directly in the vfs. In the normal case we decide what icon to use for a file in the same way (based on the mimetype). However, this happens in the gio backend, so if the backend has special needs for icons it can do whatever it wants.
This is great stuff and will allow backends to be more expressive, and applications need less code to handle icons. It does however mean that the code in Nautilus doesn’t really match the new APIs. So, since I had to completely restructure the code anyway, its with great pleasure I can announce that since yesterday NautilusIconFactory has been removed from the gio-branch, and the new icon and thumbnailing code is integrated with the NautilusFile object in a much saner way.
Most of this work is just changes needed to work with gio rather than new features. I did however add one new feature. If you resize icons so that they are larger that the thumbnail generated for them (i.e. 128 pixels) they look very blurry. I’ve seen a lot of people with such scaled up pictures of pets or loved ones on the desktop, and it looks quite ugly. So, I made the thumbnail loader try to use the file itself as thumbnail if the icon is scaled above 128 pixels:
Wax On!
Wax Off!
None of this work significantly brings down the number of gnome-vfs calls in nautilus mentioned in my last blog. We’re now down from 803 to 510 compared to 531 in last entry. I think I will need to convert the GnomeVFSVolumeManager calls to gio now, as that will bite of a large chunk of this.
The work on the nautilus-gio branch continues:
[alex@localhost nautilus-gio]$ find -name "*.c" | xargs grep gnome_vfs_ | wc -l
531
Where it originally was:
[alex@localhost nautilus]$ find -name "*.[ch]" | xargs grep gnome_vfs_ | wc -l
803
So, 33% of all gnome vfs calls have been removed.
This doesn’t seem like a lot, but much of the core of nautilus has moved over.
These are the current uses of gnome vfs in nautilus-gio:
2 gnome_vfs_async_cancel
1 gnome_vfs_async_close
1 gnome_vfs_async_load_directory_uri
1 gnome_vfs_async_open
2 gnome_vfs_async_read
2 gnome_vfs_async_set_file_info
6 gnome_vfs_async_xfer
1 gnome_vfs_check_same_fs_uris
2 gnome_vfs_close
2 gnome_vfs_connect_to_server
1 gnome_vfs_directory_list_load
1 gnome_vfs_directory_visit_uri
2 gnome_vfs_drive_compare
3 gnome_vfs_drive_eject
2 gnome_vfs_drive_get_activation_uri
1 gnome_vfs_drive_get_device_path
8 gnome_vfs_drive_get_device_type
2 gnome_vfs_drive_get_display_name
1 gnome_vfs_drive_get_icon
2 gnome_vfs_drive_get_mounted_volumes
8 gnome_vfs_drive_is_mounted
1 gnome_vfs_drive_is_user_visible
5 gnome_vfs_drive_mount
3 gnome_vfs_drive_ref
3 gnome_vfs_drive_unmount
17 gnome_vfs_drive_unref
1 gnome_vfs_error_quark
5 gnome_vfs_escape_path_string
1 gnome_vfs_escape_slashes
6 gnome_vfs_escape_string
5 gnome_vfs_file_info_list_free
8 gnome_vfs_file_info_new
1 gnome_vfs_file_info_to_gio
9 gnome_vfs_file_info_unref
2 gnome_vfs_file_type_from_g_file_type
2 gnome_vfs_file_type_to_g_file_type
9 gnome_vfs_find_directory
5 gnome_vfs_get_file_info
1 gnome_vfs_get_uri_scheme
1 gnome_vfs_get_volume_free_space
18 gnome_vfs_get_volume_monitor
2 gnome_vfs_init
1 gnome_vfs_make_directory
1 gnome_vfs_make_directory_for_uri
5 gnome_vfs_make_uri_from_input
1 gnome_vfs_make_uri_from_input_with_trailing_ws
1 gnome_vfs_make_uri_from_shell_arg
1 gnome_vfs_method_get
2 gnome_vfs_monitor_add
2 gnome_vfs_monitor_cancel
1 gnome_vfs_open
1 gnome_vfs_read
1 gnome_vfs_result_to_error
7 gnome_vfs_result_to_string
5 gnome_vfs_shutdown
4 gnome_vfs_unescape_string
4 gnome_vfs_unescape_string_for_display
2 gnome_vfs_unlink
7 gnome_vfs_uri_append_file_name
3 gnome_vfs_uri_append_string
1 gnome_vfs_uri_dup
7 gnome_vfs_uri_equal
2 gnome_vfs_uri_exists
4 gnome_vfs_uri_extract_dirname
4 gnome_vfs_uri_extract_short_name
2 gnome_vfs_uri_extract_short_path_name
7 gnome_vfs_uri_get_host_name
1 gnome_vfs_uri_get_host_port
9 gnome_vfs_uri_get_parent
2 gnome_vfs_uri_get_path
2 gnome_vfs_uri_get_scheme
3 gnome_vfs_uri_get_user_name
3 gnome_vfs_uri_has_parent
1 gnome_vfs_uri_is_local
6 gnome_vfs_uri_is_parent
1 gnome_vfs_uri_list_extract_uris
8 gnome_vfs_uri_list_free
2 gnome_vfs_uri_make_full_from_relative
54 gnome_vfs_uri_new
4 gnome_vfs_uri_ref
12 gnome_vfs_uris_match
18 gnome_vfs_uri_to_string
68 gnome_vfs_uri_unref
1 gnome_vfs_url_show_with_env
4 gnome_vfs_volume_compare
4 gnome_vfs_volume_eject
12 gnome_vfs_volume_get_activation_uri
6 gnome_vfs_volume_get_device_type
7 gnome_vfs_volume_get_display_name
2 gnome_vfs_volume_get_drive
1 gnome_vfs_volume_get_filesystem_type
5 gnome_vfs_volume_get_icon
2 gnome_vfs_volume_get_volume_type
1 gnome_vfs_volume_handles_trash
1 gnome_vfs_volume_is_read_only
6 gnome_vfs_volume_is_user_visible
2 gnome_vfs_volume_monitor_get_connected_drives
1 gnome_vfs_volume_monitor_get_drive_by_id
7 gnome_vfs_volume_monitor_get_mounted_volumes
1 gnome_vfs_volume_monitor_get_volume_by_id
5 gnome_vfs_volume_monitor_get_volume_for_path
6 gnome_vfs_volume_ref
3 gnome_vfs_volume_unmount
29 gnome_vfs_volume_unref
1 gnome_vfs_xfer_uri
A lot of these are low hanging fruit. For instance, there are 163 calls to the gnome vfs volume monitoring API, which has more or less a direct translation to GIO. Another large part is uri handling which should be translatable to GFile manipulation with some work. So it should be possible to get these numbers a lot lower fast.
However, other parts, like gnome_vfs_xfer() needs significant work. It is only called once, but is a large part of nautilus functionallity, basically doing all sorts of file copies and moves.
So, we’re going places, but we’re far from there yet.
I’ve started working on converting Nautilus to use the gio API instead of gnome-vfs. Its nowhere near done, but today I finished converting the basic file information reading and handling to gio.
Here it is, in all its glory!
Of course, it looks exactly the same as before, but its new and shiny on the inside. 
The code is available in the “nautilus-gio” branch in subversion. At the moment it only works with the standard gio local files backend. If you install gvfs it crashes due to some dbus mainloop integration conflict issues.
There was two really embarrasing bugs in the 0.1 release. First of all the argument order was switched around in the description of how to create glicks in the README. Secondly, a bug in the fuse filesystem implementation made it hang after 10 files had been opened.
So, a quick release is in order. Get your fresh new code at the glick home page.
Thanks to Stefan Westerfeld for finding these issues.
I’m back from my vacation now, and instead of spending days reading through the backlog of emails I decided to polish up glick and make a release so that people can play with it.
One problem that the initial glick version had was that all file lookups were done via the /tmp/glick_root symlink. This symlink being in /tmp and possibly being owned by someone else is a security issue. So, in the new release we instead use “/proc/self/fd/1023″ as the absolute prefix for the glick mount. While this looks a bit strange it is much more secure. However, it does make it a bit more complicated to create and test glick bundles, so glick now ships with the “glick-shell” tool that lets you point to a working directory and make /proc/self/fd/1023 point to that.
The new release also contains an easy to use script “glick-mkext2″ that creates minimally sized ext2 images from a directory, in addition to the mkglick script that creates the actual glick.
I’ve also added –icon and –desktop-file switches to mkglick that lets you embed 48×48 png icons and desktop files into the ELF file. These are stored in the “.xdg.icon.48″ and “.xdg.desktop” sections and can be easily extracted (using e.g. objdump, or some simple ELF header parsing code). In fact, glick now also ships with a tool “glick-extract” that lets you extract the filesystem, icon and desktop file parts easily.
I’ve talked to some people about the GPL licensing issue discussed in my previous entry, and I’ve come to believe that its ok to distribute glick bundles that contain non-GPL programs even though the glick code is GPL. Distributing such a bundle is really no different from distributing an iso file with both GPL and non-GPL software, which is explicitly ok due to the aggregation section in the GPL:
In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.
Of course, I am not a lawyer, so seek your own legal advice if you are unsure.
Anyway, this version of glick should be secure, easy to use and useful, please play with it and do amazing things. The source for the release and Fedora 7 rpms are availible from the glick webpage.
I’ve always admired the klik
project. At least the idea behind it, because the implementation was
(by necessity) a bit ugly. There are two main problems, first it uses
loopback mounts (requires root/setuid, has weird limitations, etc) and
secondly it does relocation of packages (so that they can find their
files) by patching the binaries.
Thats why I was so happy when i saw this blog
entry. The klik developers are working on klik2, which is working
to clean up these issues. I took a quick look at it, and they seem to
be using fuse instead of loopback, which is very good. To solve the
relocation problem they seem to be using fuse to implement a full
union filesystem.
I wasn’t completely sold on the union filesystem thing, as I thought
that was a bit overkill. All that really needs to be virtualized is one
directory, accessible via a fixed name for the running
application. Then you can install things in that given prefix, make a bundle of it and it
will run fine.
A few days later when I was lying on the beach I came up with various ways this
could be done. After a while I realized that it would be
possible to use fuse to do a runtime-less bundle system. In other
words, all you needed to install was a standard fuse, and then you can download the
app as one file and just run it. I’m on vacation and shouldn’t even be
touching my computer, but I was so exited about this idea I just had
to see if I could do it. After a day or so of hacking, i was able to make it work.
glick bundles are standard ELF files that you can run like any other
binary, but they contain all the files, libs and binaries needed for
the application in that one file. glick files embedd a filesystem in the
rodata section of the elf file, and the startup code contains a fuse
filesystem to mount this filesystem, and some magic glue code to make
it all work together.
Here are two example bundles: fancy
greeter, katach.
I’ve tested these on F7, and it seems to work. (fuse-lib and fuse must be
installed, and due to a broken Fedora packaging of fuse the user must be a
member of the fuse group or fusermount needs to be chmod:ed).
On application launch the glick code is started. It then creates a
temporary directory where the filesytem will be mounted. Then it creates a pipe to
talk to a child process and forks. The child process starts up the
fuse filesystem (reading from the automatically mmap:ed rodata section) and mounts that on the temp directory. Then it talks
to the parent by writing to the pipe.
The parent code opens the directory, and uses dup2() to store the file
descriptor to it as fd number 1023. Then it creates (or ensures that
there already exists) a symlink called “/tmp/glick_root” that points
to “/proc/self/fd/1023″. This is the magic part. Since we have the
target directory opened at fd 1023 the /tmp/glick_root symlink will
point to the mounted fuse filesystem for this process and all its children
(that don’t close fd 1023). This means the code in the filesystem can
hardcode /tmp/glick_root to access their files. (Having this in /tmp
is slightly non-ideal. A better place would be in say /var/glick_root,
but then it wouldn’t be a runtime-less system).
Then it reads from the pipe, waiting for the mount to finish. When the
mount is up it it execs /tmp/glick_root/start which is a script or a
symlink that starts up the real application inside the fuse mount.
When the application dies the pipe to the fuse child is automatically
closed. The child notices this, unmounts the mount, removes the
temporary mountpoint and exits.
The implementation uses e2fsprogs-libs to implement the fuse
filesystem (based on fuse-ext2 by Jeff Garzik). This library is
unfortunately GPL, which is not ideal for maximal reuse (my code in glick is X-style
BSD). Does embedding an ext2 filesystem inside an elf
file that contains GPL code make the files in the filesystem a derived
work of the GPL code, or is this a “mere aggregation” in the terms of
the GPL? License questions abound… Anyway, perhaps using fuse-cramfs
that the klik people are working on is a better idea anyway.
The code is availible via git here. Play
around with it, experiment, have fun. I did.
Note: The name “glick” is a pun on the kde:ish name of the the klik
project. This is not ment to be a slight or anything like that. I
greatly appreciate the work of the klik project (although I do believe
that their choice of name unfortunately is costing them mind-share
amongs gnome users). Hopefully the klik people will look at my work and steal the appropriate ideas.
Bad Behavior has blocked 203 access attempts in the last 7 days.