Last week I landed a new feature in Glib that I’ve been wanting to do for a long time: A resource framework. Resources are things that are naturally part of an application or library, but not really normal code. For instance, our code increasingly uses xml to describe user interfaces and menus.
Traditionally these either had to be manually inserted into the code, like so:
static const gchar *ui_info = "<ui>" " <menubar name='MenuBar'>" " <menu action='FileMenu'>" " <menuitem action='Quit'/>" " </menu>" " </menubar>" "</ui>";
Or a file was stored in /usr/share/$application/ and you had to write code to manually find and load the file, and cache it if used often. This is not a lot of code, but it can be tricky as all I/O code needs to handle errors and the external file makes it harder to make the library/app relocatable.
Instead, with resources you store your data file as plain files in your source tree, edit them with your favourite editor, with full syntax highlighting, automatic indentation, etc. Then you reference these files from a resource description file (another xml file) and use glib-compile-resources to compile these into a binary chunk which is linked into the application.
The resource framework then automatically registers all such resource bundles in a global namespace, where you can quickly look up resource data by a resource path. There are API calls to get the data as a direct pointer, as well as streaming data, but in most cases there are helper functions that let you just specify the pathname. So, for instance, you can just do:
gtk_builder_add_from_resource (builder, "/org/gnome/appname/menu.ui", &error);
Which would handle all the work for you. And while this looks like an I/O operation its really just a hashtable lookup on linked-in data, reading from the (shared, readonly) data section in your executable, so its very fast and safe.
Additionally there are some tricks the resource compiler can do for you. For instance, you can specify that a resource should be compressed, which means that the data is stored compressed, and the APIs uncompress for you automatically. You can also specify that xml files should be pre-processed to strip away whitespace, which avoids wasting time and memory on something that is not useful at runtime.
There is also support for resource:// URIs, which means you can easily reference resource data like icons from e.g. CSS and UI files.
On Linux we use some gcc specific extensions to store the resources in separate ELF sections, which means its very easy to extract resource data from the binaries. Glib even ships with a tool that lets you do this:
$ gresource list libgtk-3.so /org/gtk/libgtk/cursor/dnd-ask.png /org/gtk/libgtk/cursor/dnd-copy.png /org/gtk/libgtk/cursor/dnd-link.png /org/gtk/libgtk/cursor/dnd-move.png /org/gtk/libgtk/cursor/dnd-none.png /org/gtk/libgtk/gtk-default.css /org/gtk/libgtk/gtk-win32.css $ gresource extract libgtk-3.so /org/gtk/libgtk/gtk-default.css @define-color fg_color #000; @define-color bg_color #dcdad5; @define-color text_color #000; ...
If you’re interested in using resources in your application, check out the documentation, or look at this example commit that converts Nautilus to use resources.
Alex, using gcc constructors for this is a really bad idea, are you sure you got everything right regarding the order in which these constructors are run, and in regards to this constructors when they are used in shared libraries or loadable modules, or even shared libraries which are pulled in by loadable modules?
Using gcc constructors is usually not a solution but a problem in itself.
Lennart: The use of gcc constructors is optional, we also support a manual initialization mode, although that requires your code to have some initialization point, which is not always true for libraries.
As for the ordering issues, I don’t think its really a problem, as the constructors themselves are so trivial. All they do is allocate a chunk of memory and put it on a list. So, there are no actual ordering requirements.
Also, having active code for the registration of the resources might result in quite a hit on performance at startup if a gazillion of small objects are registered by your app and all its shared object.
I think it’s really important to turn this “active” registration into something passive. For example, glibc provides dl_iterate_phdr() which should be enough to implement this entirely passively (with a tiny bit of ELF parsing).
The thing about constructors/destructors is that they appear trivial in their code but seldom are in their effect. And if I hear that you allocate memory in them this causes all my red lights to flash, because this either causes memory leaks if your .so is pulled in via dlopen() (or indirectly via dlopen()), or might result in code not expecting it to access invalid memory. The least you need to do is ensure that shared objects which do this become unloadable…
Lennart,
there are destructors to go along with the constructors we use. We are not stupid…
Very cool. I’ve used resources in c#, which was really handy. Glad to see it is now simple to use in c
Out of curiosity, why is GResource not a GObject? Is it not intended to be used by language bindings?
Tristan: Because they have to be allocated very early in the constructors, before g_type_init() has been called.
Also, GResource as is isn’t particularly important API for app, its mostly used by the autogenerated APIs.
Lennart: Not only do we have destructors to go with the constructors, we also have test cases for runtime module loading and unloading.
I don’t think using dl_iterate_phdr is a solution, as it only works on glibc, meaning that instead of actually solving any problems we have we just punt them to other platforms.
Also, I don’t think a gazillion objecs are registered. We just register a single pointer to a binary blob. I.E it doesn’t actually try to look into the resource blob at registration time. And this only happens once for each shared library that uses resources (which is not gonna be all of them).
Great stuff! It makes me wonder how easy we could have static self-contained application.
Btw, the last post was about Windows support, so how is GResource supported on other platforms? Is it a Linux only thing atm?
Marc-André:
Self contained applications is a different matter. Resources is not for that. There is no way you can e.g. include another library as a resource, nor is it a good idea to include things other processes needs to read, like docs or desktop files.
If you’re interested in application bundles, see my other work on glick2 for that kind of thing.
There is nothing linux specific about resources, they work fine on Windows, although the gresource tool will not be able to extract resources from win32 dlls.
Alex, this is a great feature!
do you include the functionality to convert png files to in-line pixbufs in the same code so we can handle that the same way as other resources? Right now in Bluefish we convert all icons to C code that is compiled into the binary, so it would be great if there would be a single solution for all these issues.
b.t.w. is there a nice way for backwards compatibility? I would like to keep Bluefish compatible with gtk-2.24 for a while, but at the same time use the new resource system if available.
Oliver:
There is no support for inline pixbufs atm. Its slighly problematic to do in the resource compiler, as it is in glib and doesn’t depend on gdk-pixbuf. It should be possible to do using an external tool though.
GResource is a glib api, and as such works fine in Gtk2. However, the gtk+ helper functions were added in gtk3 only, so you’d have to use a bit more code to actually use the resources.
Olivier:
I added support for the to-pixdata preprocessing directive in resource files.. That means you can get gdk-pixbuf-csource pixbuf data sharing when using gdk_pixbuf_new_from_resource().
Mathias, Alex: the main problem actually comes from the use of the destructors, not the use of the constructors!
Alex, dl_iterate_phdr() actually originates from Solaris afaik, and is also available on the BSD. Should be more portable than GCC constructors/destructors actually.
Lennart:
Solaris and BSD are easy, being (mostly) ELF based, but for platforms like OSX (Mach-O) and Windows (PE) its much harder.
I know that constructor/destructor ordering is generally problematic, but I don’t see how it would be a problem for this.
Here is more or less the entire code used by the constructors/destructors. What kind of issues do you expect from it?
(Note: The constructor macros we use are only availible on some platforms, and we ensure that they actually work sanely for e.g. module load/unload case.)
/* In libglib.so */
GResource *
g_resource_new_from_data (GBytes *data,
GError **error)
{
GResource *resource;
GvdbTable *table;
table = gvdb_table_new_from_data (g_bytes_get_data (data, NULL),
g_bytes_get_size (data),
TRUE,
g_bytes_ref (data),
(GvdbRefFunc)g_bytes_ref,
(GDestroyNotify)g_bytes_unref,
error);
if (table == NULL)
return NULL;
resource = g_new0 (GResource, 1);
resource->ref_count = 1;
resource->table = table;
return resource;
}
void
g_resources_register (GResource *resource)
{
g_rw_lock_writer_lock (&resources_lock);
registered_resources = g_list_prepend (registered_resources,
g_resource_ref (resource));
g_rw_lock_writer_unlock (&resources_lock);
}
void
g_resources_unregister (GResource *resource)
{
g_rw_lock_writer_lock (&resources_lock);
if (g_list_find (registered_resources, resource) == NULL)
{
g_warning (“Tried to remove not registred resource”);
}
else
{
registered_resources = g_list_remove (registered_resources,
resource);
g_resource_unref (resource);
}
g_rw_lock_writer_unlock (&resources_lock);
}
/* Linked into App/lib */
GResource *contacts_resource = NULL;
static void constructor (void)
{
if (contacts_resource == NULL)
{
GBytes *bytes = g_bytes_new_static (contacts_resource_data.data, sizeof (contacts_resource_data.data));
contacts_resource = g_resource_new_from_data (bytes, NULL);
if (contacts_resource)
g_resources_register (contacts_resource);
g_bytes_unref (bytes);
}
}
static void resource_destructor (void)
{
if (contacts_resource)
{
g_resources_unregister (contacts_resource);
g_resource_unref (contacts_resource);
contacts_resource = NULL;
}
}
If the resource code can call g_new() from a global constructor, then how can an application ensure that it calls g_mem_set_vtable() before any other glib functions are called?
muppet:
True, that is an issue.
We should switch GResource to use g_slice_new instead. I think that will remove all the g_mallocs on the resource constructor path.
Actually, that doesn’t work as g_slice uses g_malloc to get its memory…
I made some changes in glib and now the constructors don’t allocate memory (the actual resources are lazily allocated).
A very interesting post for developers. It helps to understand how the low level works.
I have a questions when you say:
“For instance, you can specify that a resource should be compressed, which means that the data is stored compressed, and the APIs uncompress for you automatically.”
What compression/uncompress method use the API?
shakaran:
We use zlib, as this is what glib currently requires. I don’t want to add further complex dependencies for something that might not make a very large difference.
why the API is so confusing? sometimes g_resource* and sometimes g_resources*?
Mike: They are two different things, g_resources_* works on the set of globally installed modules and is what you’d normally use.
In fact, you probably wouldn’t even use that, but rather things like gdk_pixbuf_new_from_resource() or gtk_builder_add_from_resource().