Nautilus memory use analysis

Yesterday I hacked up a glib patch to track memory
use. Its a g_malloc replacement that has some memory type tracking features which are nicely integrated with common glib/gtk+ types.
All the app has to do is call g_mem_set_vtable(glib_mem_tracker_table)
the first thing you do in main, and then calling g_mem_track() will
write the full allocation information into a file called
“memtrack.dat” in the current directory. You can then use a separate
app to analyze this file at your leasure.

So, what does this give us for Nautilus? I started it, browsed
around a bit and ended with a window showing my homedir (which has about 600 files), then I triggered the memory dump. The
memory use summary is now:

./analyze-memdump -s nautilus-run-with-home.dat 
total size: 5201628
 unknown type: 1609963
 strings: 914293
 objects: 495264
 slice: 559180
 list headers: 165400
 pixbuf data: 1457528

So, we’re using about 5 megs of heap. Almost 1 meg of this is
strings, 1.5 meg is pixbufs. Lets look at what sort of strings we have:

./analyze-memdump -z nautilus-run-with-home.dat
file nautilus-file.c: 254163 bytes
file pango-layout.c: 180279 bytes
file gfileutils.c: 111728 bytes
file gstrfuncs.c: 63963 bytes
file gunicollate.c: 40120 bytes
file gnome-vfs-file-info.c: 32965 bytes
file gtkicontheme.c: 29314 bytes
file gnome-vfs-mime-info-cache.c: 24787 bytes
file gtkrc.c: 21428 bytes
file gconf-value.c: 20343 bytes
file gdataset.c: 20222 bytes
file nautilus-icon-canvas-item.c: 12314 bytes
...
string of size 64: 2020 (129280 bytes)
string of size 2048: 53 (108544 bytes)
string of size 1024: 106 (108544 bytes)
string of size 128: 147 (18816 bytes)
string of size 20: 868 (17360 bytes)
string of size 512: 33 (16896 bytes)
string of size 19: 869 (16511 bytes)
string of size 18: 839 (15102 bytes)
string of size 11: 1340 (14740 bytes)
string of size 60: 238 (14280 bytes)
string of size 16: 844 (13504 bytes)
string of size 14: 923 (12922 bytes)
...

Lots of strings in nautilus-file.c, lets look into that file:

 ./analyze-memdump -f nautilus-file.c nautilus-run-with-home.dat 
For the file 'nautilus-file.c':
 total size: 254163
  unknown type: 0
  strings: 254163
  objects: 0
  slice: 0
  list: 0
  pixbuf data: 0
block size 2048: 53 allocations (108544 bytes)
block size 1024: 106 allocations (108544 bytes)
block size 512: 32 allocations (16384 bytes)
block size 256: 7 allocations (1792 bytes)
block size 128: 4 allocations (512 bytes)
block size 64: 9 allocations (576 bytes)
block size 54: 1 allocations (54 bytes)
block size 48: 1 allocations (48 bytes)
line 2637: 5 bytes (1 allocations)
line 2675: 9 bytes (1 allocations)
line 4731: 32 bytes (2 allocations)
line 221: 37 bytes (8 allocations)
line 1397: 76 bytes (4 allocations)
line 284: 211 bytes (6 allocations)
line 1437: 17333 bytes (1001 allocations)
line 6063: 236460 bytes (217 allocations)

A lot of large strings, and they seem to all be from line 6063 of
nautilus-file.c. Looking at this code reveals that this is the memory
used for storing the first bit of text files so that we can show the
text in the icon. We only save the text in the first 80×24 chars of
the file, but perhaps we could save less of it and reload the data for
the uncommon case of a large icon that needs more text.

The string use in pango-layout.c isn’t suprising, as Nautilus stores
the PangoLayouts for each label in the icon view to be able to draw it
quickly without having to re-layout. Unfortunately PangoLayout also
stores a copy of the actual strings, which we don’t really need.

Looking back at the totals we have 0.5 meg of GObjects (although this
doesn’t include memory allocated by the objects). Lets have a look at
that:

./analyze-memdump -o nautilus-run-with-home.dat 
type 'NautilusVFSFile': 121080 bytes, 1009 objects
type 'PangoLayout': 88060 bytes, 1295 objects
type 'NautilusIconCanvasItem': 32256 bytes, 504 objects
type 'GtkAccelLabel': 28880 bytes, 190 objects
type 'GstPadTemplate': 27512 bytes, 362 objects
type 'PangoCairoFcFont': 24864 bytes, 111 objects
type 'GstElementFactory': 23856 bytes, 213 objects
type 'GtkImage': 15808 bytes, 152 objects
type 'GtkImageMenuItem': 15100 bytes, 151 objects
type 'GtkSeparatorMenuItem': 15072 bytes, 157 objects
type 'ClearlooksStyle': 14388 bytes, 11 objects
type 'GdkWindowImplX11': 9100 bytes, 91 objects
type 'GdkPixbuf': 8320 bytes, 160 objects
type 'GdkWindow': 8008 bytes, 91 objects
type 'GtkAction': 7208 bytes, 106 objects
type 'GtkRcStyle': 6640 bytes, 20 objects
...

Lots of NautilusFiles, PangoLayouts and icons which is hardly
surprising. However, we also have a lot of accel-labels, and menu
items. This is from the menu in the window and the hidden menu in the
desktop window. I wonder what the gstreamer objects are from though.

Another way to look at memory use is to look at what files allocate a
lot of memory:

./analyze-memdump -p nautilus-run-with-home.dat 
Per-file memory use:
No file info: 201536
file 'gdk-pixbuf.c': 1457837
file 'gtype.c': 606802
file 'glyphstring.c': 453448
file 'ghash.c': 305840
file 'pango-layout.c': 274576
file 'nautilus-file.c': 254163
file 'gnome-vfs-file-info.c': 146309
file 'glist.c': 131400
file 'gfileutils.c': 124571
file 'gtkicontheme.c': 97906
file 'nautilus-icon-canvas-item.c': 97538
file 'gclosure.c': 75316
file 'gstrfuncs.c': 71390
file 'gsignal.c': 69152
file 'gdataset.c': 63758
file 'garray.c': 63104
file 'fonts.c': 49232
file '../glib/gbsearcharray.h': 48972
file 'pango-item.c': 44640
...

Here we also see the PangoLayouts using a lot of memory. Not only in
pango-layout.c (which is the strings), but also in glyphstring.c which
is the data from the PangoLayouts that I’m actually interested in. It
seems to be about twice the size as the strings though, so its not
very compactly stored. It would be interesting if one could take a
layouted PangoLayout and turn it into a lean platform-specific render
object that you could only render, and nothing else. That way i could
throw away the PangoLayout when it was done measuring the string. It
could also possibly store the glyph strings in a more compact way
(PangoGlyphInfo looks a bit large). Or maybe I should just bite the bullet and re-layout when drawing?

To test this stuff you have to apply these patches for glib and
gtk,
and then you have to rebuild your app and all glib-using libraries it
uses. The analyzer app is
here.

If you are interested in Nautilus you can download the

memory dump
I used and play with it. It should work on any 32bit little-endian system.

Lisp pointer size?

Today I got this from emacs:

Warning: memory in use exceeds lisp pointer size
Killing some buffers may delay running out of memory.
However, certainly by the time you receive the 95% warning,
you should clean up, kill this Emacs, and start a new one.

I’ve never seen that before. The process was just 126 megs and about two months old.

Gtk+ printing on windows

This week I have continued to work on the highlevel platform independent Gtk+ printing API (availible in libegg in cvs). Its getting pretty good by now, and I have a couple of test apps that verify the API works.

I spend most of the time this week in windows, implementing the win32 version of the backends. I did that for two reasons, first of all any Gtk+ API has to be portable, so I have to make sure that the API we come up with is implementable on windows. The second reason is that windows already have a print dialog, and working with it lets me learn a lot about the general requirements of printing dialogs that we can use as input both for the highlevel API and later when implementing the unix dialog.

I’ve put up a small windows demo
here.
Its got all the dlls required to run it. Just unzip it to a directory and click on start.bat to launch it. (It spews error output sometimes because it can’t find the stock icons, I dunno why that is.)

Print yourself!

I’ve been working a bit on a printing API for gtk+. Its a highlevel API that integrates cairo, pango and a native print dialogs. Its really easy to get nice printing output, and it will work on all platforms, using native printing. I just made a code drop so that people can look at the APIs and comment on it.

The example code in it is pretty awesome, it actually prints itself. Here is the output pdf.

If you read that pdf you’ll see it produces some other output too. I’m sure interested parties can find that file too. :)

At last!

Daniel Veillard built the ekiga beta (ekiga is basically GnomeMeeting 2.0) in rawhide yesterday, so I decided to try it out. For the longest time I’ve thought GnomeMeeting was a really cool idea, but whenever I tried to actually use it I’d alway fail due to some firewall issue. (I have a pretty standard setup with a Linksys WRT54 broadband router as firewall.)

Not so this time!

It was incredibly smooth to set up, even autodetecting how to avoid NAT problems. The only manual thing I had to do was to enable the top secret “+20dB mic boost” setting in the volume control. I tried calling various echo servers on the net, which worked fine. Then I called caolan, who just got a “commercial VOIP thingie”. Worked immediately! Maybe the firewall handling got better, or maybe SIP is just easier to get through firewalls. I don’t care, and I don’t have to, it just works!

The integration with the free ekiga.net service and its central address book really makes this much more usable. Hats of to Damien!

Seek and Ye Shall Find

The last two weeks I have been working on the nautilus-search (and now nautilus-search2) branch of Nautilus. The initial code in this branch was written by andersca and was later maintained by joe. It had a pluggable search interface with a beagle implementation that let you easily search for files from Nautilus. Places->Search in the menu would give you a text entry in the toolbar, and when you typed text there the matches was shown in the normal directory view. Here is how it looked:

While pretty cool and useful, this really isn’t state of the art compared to e.g. MacOS X or MS Vista. For one, it doesn’t support what Apple calls “Smart Folders” and MS “Virtual Folders”, nor does it allow you to specify the search other than with text. Furthermore, the search acts as a special type of folder, but I didn’t like the fact that its not obvious that this is not a “normal” folder.

So, after spending some time on the code I now have this:

The blue area is a special part of the window I called “extra location widgets” in the code. It can be used to show that this is a special kind of folder, and I think we should use it for folders in burn:/// also. Furthermore, the “+” button lets you add limitations on the search:

You can also search by type:

When you are satisfied with your search you can use File -> Save Search As… which gives you this dialog:

Looking in home, we see the newly saved file “search for alex.savedSearch”. Its really a simple xml file describing the query. However, we can tell by the expander that Nautilus treats it as a folder.

If you expand it, you get the results of the search as if it was a normal folder:

And if you click on it you show the file as a folder:

You can also edit it by clicking on the edit button, giving you back the original query edit widgets. When you’ve changed them to your liking, File -> Save Search updates the search file on disk:

All this code is on the nautilus-search2 branch in cvs. Its mostly working, I just need to fix some bugs and add a simple (non-indexing) search backend and I think its ready to be merged to HEAD.

Sabayon LDAP support

Earlier this week I blogged about Sabayon, the profile editor. I got several comments about how it would be nice if it supported LDAP. So, I spent some time this week hacking up LDAP support.

I wrote some initial documentation on how this works, and now I’d like some feedback. The system is very flexible and IMHO nice, but I’ve never administrated an LDAP network, so I’m not the best person to ask.

If you are interested in Sabayon and LDAP, please read the docs and send your impressions to the mailing list.

Live, uncut french egg-sauce thingie

The last couple of weeks I have been working on Sabayon, an application that some other Red Hat people started on earlier this year. Its an editor for user profiles, where you can set up settings and add files for a class of users and easily push them out to users. Today I released version 2.12.1, which has a bunch of fixes and new features.

The interesting thing about Sabayon is the way you edit the profile. Instead of using a completely new interface it uses the standard interface you normally use. This is accomplished by launching an actual desktop inside a window, letting you make your changes in that desktop and then save it. You can also look at the changes that has happened in the session and ignore some of them, or make some of them mandatory.

Just describing Sabayon isn’t doing it justice though. To really see how this is different from other similar things you have to see it in action. So, I prepared a screencast
of it.

Once you’ve finished editing your profile it will be stored as a zip file with xml metadata that can easily be applied to a user by running the sabayon-apply tool. There is also support for deploying the profiles from a central http server.

I think this is a really cool app, but I’m a programmer, not really an administrator. I wonder what system administrators think of this? Is it an interesting approach? Is it better than what you’re using currently? Would you use this?

If you want to test Sabayon I recommend you use version 2.12.1. Its currently building in Fedora extras, so it should be available soon.

Wobbly widgets

Recently I’ve been researching a bit into requirements and usecases for a new canvas for Gnome/Gtk+. One of the important features of a canvas is the ability to embedd normal Gtk+ widgets into the canvas and have them integrate nicely. This means the embedded widgets should properly stack with other canvas items, and (less importantly) they should support arbitrary transformations.

The problem with this is that widgets in Gtk+ use child X windows, for clipping/scrolling and to get events. These windows are managed by X and won’t correctly mix with the drawing of the other items in the canvas. This can be seen in the widget item in gnome-canvas, which just doesn’t work very well.

An obvious solution to this is to render the whole widget to an offscreen pixmap and then just copy that translated to the canvas when rendering the canvas item. However, currently there isn’t a way to render Gtk+ widgets to a pixmap. There are some hacks you can use, but they don’t work at all for widgets that have child windows.

This week I’ve been working on a change to Gdk that will allow it to take any GdkWindow and make it (and its children) into an offscreen pixmap, much like the Composite Extension but client-side. Once we have this you can do all sorts of effects with widgets, just like Composite allows effects on a larger scale.

Yesterday I managed to get it to render an actual widget tree for the first time, so I hacked up a simple demo to show off the sort of things you can do with this:

When seen live the water effect actually moves, as can be seen here.

This work is not nearly finished, I haven’t even started to think about how do handle input events, but it does look promising.