Category: Gnome

Someone threw a 8-million cell csv file at Gnumeric. We handle it, but barely. Barely is better than LibreOffice and Excel if you don’t mind that it takes 10 minutes to load. And if you have lots of memory.

I looked at the memory consumption and, quite surprisingly, the GHashTable that we use for cells in sheet is on top of the list: a GHashTable with 8 million entries uses 600MB!

Here’s why:

• We are on a 64-bit platform so each GHashNode takes 24 bytes including four bytes of padding.
• At a little less than 8 million entries the hash table is resized to about 16 million entries.
• While resizing, we therefore have 24 million GHashNodes around.
• 24*24000000 is around 600M, all of which is being accessed during the resize.

So what can be done about it? Here are a few things:

• Some GHashTables have identical keys and values. For such tables, there’s no need to store both.
• If the hash function is cheap, there’s no need to keep the hash values around. This gets a little tricky with the unused/tombstone special pseudo-hash values used by the current implementation. It can be done, though.

I wrote a proof of concept which skips things like key destructors because I don’t need them. This uses 1/3 the memory of GHashTable. It might be possible to lower this further if an in-place resize algorithm could be worked out.

I have been known to complain loudly when I see code that I feel should have been better before seeing the light of day. But what about my own code? Divinely inspired and bug free from day one? Not a chance!

With Gnumeric as the example, here is what we do to keep the bug count down.

• Testing for past classes of errors. For example, we found errors in Gnumeric’s function help texts, such as referring to arguments that do not exist or not describing all the arguments. The solution was not only to fix the problems we found, but also to write a test that checks all the function help texts for this kind of errors. Sure enough, there were several more. They are gone now, and new ones will not creep in. We do not like to make the same mistake twice!
• Use static code checkers. This means that we keep the warning count from “gcc -Wall” down so know nothing serious is being ignored. We have looked at c-lang and Coverity output and fixing the apparent problems. (Those tools have pretty high false report rates, though.) We occasionally use sparse too and have a handful of dumb perl scripts looking for things like GObject destroy/finalize/etc handlers that fail to chain up to the parent class.
• Use run-time code checkers. Gnumeric has been run through Valgrind and Purify any numbers of times. It is part of the test suite, so it happens regularly. This is regrettably getting harder because newer versions of Gtk+ and the libraries upon which it is built hold on to more and more memory with no way of forcing release. Glib has a built-in checker for some memory problems. We use that too.
• Automated tests of as many part of the program as we have found time to write. The key word here is “automated”. I used to be somewhat scared of changing the format string (number rendering) code, because there was basically no way of making sure no new errors were introduced in that hairy piece of code. With the extensive test suite, I have no such reservations anymore.
• Fuzzing, i.e., deliberately throwing garbled input at the program. I wrote tools to do this subtly for xml and files inside a zip archive in such a way that the files are still syntactically correct xml or zip files — otherwise you end up only testing the xml/zip parser which is fine, but not sufficient.
• Google for Gnumeric. Not every will report problems to us, but they might discuss issues with others. Google seems to be pretty good at finding such occurrences.

The take-home message from this is that code quality is work. Lots of work. And yet we still let mistakes through. I blame that on the lack of a proper QA department.

I my optics, computers are here to get certain jobs done. That means it is all about applications, not eye candy: bouncing icons, themes, semi-transparent windows. My real-life work desk is not transparent, and I do not use semi-transparent paper.

Producing large applications is a lot of work, so when I write a piece of (hopefully) well-designed code, I want that code to stay written. I do not want next week’s GTK+ deprecation to come along and, effectively, cause my code to bitrot. (and I really do not want to write two different pieces of code for the job: one for “old” GTK+ and one for “new” GTK+.)

Moving from GTK+ 1.x to GTK+ 2.x was painful. I do not need anything like that again. Talks about breaking API every 3-4 years and advice like “Stay up to date, adapt your application code early” (and, by implication, often) is a clear indication that keeping applications running is likely to mean spending much time cleaning up after someone with an attention span of a few years.

Maintaining code like GTK+ is not hard. Calling it hard because you want to play with some new toy is deceiving.  Maintaining can be tedious, but if you do not want to maintain, please do not start writing new GTK+ code. You will surely abandon that prematurely too, so you have no business writing library code. Instead, go write a useful application: if you abandon that, I probably do not have to care.

In a previous post, I showed how a GTK+ theme engine can corrupt memory of any application unfortunate enough to be used with it.

In today’s edition, our guest star is the Qt theme engine. It does not, as far as I know, corrupt your memory or otherwise make your innocent application crash.[*] Instead it changes how your program works. For example, for Gnumeric it changes how numbers imported are handled.

If you import the number “8,5” in a decimal-comma locale then you would hope to get eight-and-a-half, right? Well, with the Qt theme you get eight and we, the Gnumeric team, look incompetent. The problem arises because the Qt theme, quite reasonably, initializes the qt library. During that, less reasonably, the following code gets executed:

setlocale( LC_ALL, “” ); // use correct char set mapping
setlocale( LC_NUMERIC, “C” ); // make sprintf()/scanf() work

I am not kidding. The Qt library thinks it should change your locale. What on Earth have the Trolls been drinking? Impure home destilled booze in large quantities?

This problem in various disguises have had us puzzled for quite a while and only very recently was the Qt theme identified as the triggering factor. Once that happened, it was not too hard to locate, but before that we have spent maybe 40 hours looking for this bug. The workaround is to set up a one-shot idle handler that resets the locale properly when the gui comes us. (Repeat this for every GTK+ program that displays or accepts floating-point values.)

The Qt theme people never caught this. If they are mostly “theme” people I can understand, but if they are mostly “Qt” people they really should have known. In either case, it is another exhibit for the case that the GTK+ theme model is seriously flawed.

[*] Well, if you use threads it might. The Qt library calls setlocale to change locale and that’s not allowed in a threaded program.