Archive for the ‘General’ Category

I’m looking at you

Friday, September 12th, 2014

Hi,

I write to you all today on a solemn matter, one which I fear will be forgotten and ignored if nobody starts some discussion on this.

Earlier this week, some of you may have noticed that for a very short time there was a rather angry post by Philip Van Hoof, he sounded quite frustrated and disturbed and the title of his post basically said to please remove him from the Planet GNOME feeds.

Unfortunately this blog post was even deleted from his own blog, so there is nothing to refer to here, also it was gone so fast that I have a hunch many Planet GNOME readers did not get a chance to see what was going on.

What I want to highlight in this post is not this frustrated angry post by Philip, but rather the precursor which seems to have led us to this sad turn of events.

Let’s make things better

In late May this summer, Philip submitted the post “Let’s make things better“. This post is also deleted from his own blog, I’m not sure for what reasons, I’m keeping the link alive here incase Philip feels inspired enough to at reactivate that post (it would help for people to see this in perspective, as people who have not read that entry may suspect it contained rudeness or bad language or sweeping accusations or something, which simple was not the case at all).

Yes, a lot of you readers know about that post, many of you would probably prefer I don’t bring it up, but the problem is that many people just don’t know what happened. Also the result of him deleting his post is that people don’t get any chance to verify these false claims of indecency which were aimed towards him for writing a very sensible post.

What I can say, is that the post did not use any distasteful language, he was not rude and did not single anyone out or blame anyone, he just said some really sensible things which happened to annoy a certain few members of our community.

I think the critical part which made people react irrationally to his prose, ran something like:

Maybe if we spent a little less time on outreach, and a little bit more on development…

And went on from there, he was basically arguing that our efforts on sustaining programs such as OPW are not a part of our mission, and that maybe our attention would be better spent writing excellent software (I’ll be happy if the post re-appears so people can read it in it’s integrity, as I don’t have a copy anymore).

I think, given the turn of events, this recent post by Philip requesting to be removed was a final attempt to try to do something good for a community that just keeps telling him that his views are wrong, dirty, and need to be censored, i.e. he got a lot of flak from the community at large for absolutely no good reason at all – if anyone needs to be ashamed, it’s us, as a community, for failing him.

I’m looking at you

It’s generally bad form to name people in public, however the wider GNOME community needs to know what is really going on in this case and they will not have the evidence to judge for themselves without references. That said these are only a couple of excerpts from the circus of public shaming which followed Philip’s perfectly reasonable blog post.

 

Paolo Borelli makes a response to someone who quoted Philip’s blog in a positive light on a public mailing list, and he goes out of his way to mention his public opposition to Philip speaking his mind:

However you also started off by citing Philip’s blog post and honestly I found that post wrong and disturbing

Taken in context of the mail thread, it looks as though the original poster is to be considered lucky to be taken seriously in any measure, just for referring to the said blog post which puts a little scrutany on our GNOME identity as an outreach foundation.

Paolo, really ? I would never have expected this behavior, do you really feel it’s necessary to call Philip’s call to reevaluate our position on these matter as “wrong and disturbing” ?

We have a long history you and I, I thought I knew you better than that.

 

Alberto Ruiz takes it a step further, again taking a public stance against Philip:

“I’ve been asked to remove your blog by several people and I’ve reached the conclusion that it would be a really bad idea because
it would set the wrong precedence and it would shift the discussion to the wrong topic (censorship yadda yadda). Questioning OPW should be allowed.

The problem with your post is that if not questioned by other people (as many have done already) it would send the wrong message to the public and prospect GSoC, OPW and general contributors. Your blog was the wrong place to question and your wording makes it clear that you have misunderstandings about how the community works.”

Alberto, I’m disappointed in you. There is no censorship on Planet GNOME, you know that, I know that, and asides from one silly “upskirt” incident in the history of Planet GNOME, this has never caused any issues.

Moreover, it is simply not your call, or anyone’s call to make, to decide that a long time member of our community’s politely and consicely formed opinion be censored from Planet GNOME just because it disagrees with what some of the other members think.

It is not your call to say that people should not be questioning things on Planet GNOME, especially since that is EXACTLY where it will be heard. Have you considered that he takes this issue very seriously and has decided, as is his right, to raise the matter for open public discussion ? Public discussion on the direction of GNOME is what we do in GNOME, we are the foundation and contributors and public discussion needs to happen about critical matters in order for us, the public, to make good decisions about the future of GNOME.

 

Finally, Emmanuele Bassi, I know his recent post was pretty “out there”, anyone would expect him to be frustrated after the treatment this community has given him, the public shaming and insolence this community has shown him by taking such an opposed stance against his expressing himself would be enough to drive anyone nuts.

Don’t you think, though, that his post was a last-effort attempt to be heard and be a positive influence for change in GNOME ?

Do you really think this immediate response to a frustrated blog post was the correct way to diffuse the situation ?

Really, we should do better to protect our own, Philip obviously had a rough time in the last couple months, his blog post was not an excuse to quickly sweep him under the rug, but a challenge to call people to action and actually openly discuss change.

If we don’t have people like Philip who are at least willing to fight for our ability to openly discuss things, then I fear the worst for this community in the long run.

Moral of the story guys… Please get a grip, I’m really not impressed with how people have responded to Philip this summer, it could have equally been any of you, and if you had something important to share, I would be equally disappointed if the community had so aggressively shouted you down.

And no, I was never a proponent of the CoC effort, but please guys at least try to remember the first rule: Assume that others mean well.

All the best.

 

Amendment

Today someone pointed out that since the original post at the end of may is missing, noone can form an opinion of their own. I did not have access to it at the time but another commentor was kind enough to paste a copy:

Matthew gets that developers need good equipment.
 
Glade, Scaffolding (DevStudio), Scintilla & GtkSourceView, Devhelp, gnome-build and Anjuta also got it earlier.
 
I think with GNOME’s focus on this and a bit less on woman outreach programs; this year we could make a difference.
 
Luckily our code is that good that it can be reused for what is relevant today.
 
It’s all about what we focus on.
 
Can we please now go back at making software?
 
ps. I’ve been diving in Croatia. Trogir. It was fantastic. I have some new reserves in my mental system.
 
ps. Although we’re very different I have a lot of respect for your point of view, Matthew.

Application Bundles Revisited

Saturday, December 14th, 2013

This is a follow up post on the one I made earlier this week on Glade application bundles.

What I had hoped to be a simple bundling experience turned out to actually take me all week to complete, but hey, you get what you pay for. And I’m very happy with the outcome so it was well worth the effort.

The new test bundle can be found here, and I’m very proud to announce that we require exactly the 2.7 ABI form glibc.

We still require fontconfig/freetype/Xlib libraries to be present on the target host to run what is inside the bundle, and we still require libz and a moderately old version of libglib to actually unpack the bundle.

For the AppImageKit glib & zlib dependencies, I plan to make those static dependencies so that they will not even be required, but first I must do battle with CMake to make that happen.

 

How can I use this to bundle my GTK+ application ?

This is a question I’ve been expecting to hear (and have been hearing). As the last iteration was only a spike, it involved a very ‘LFS’ technique of manually preparing the bundle, but that has been fixed this week and now you can use the bundling mechanism with a collection of customized jhbuild scripts.

Here is an outline of what we have in our build/linux/ directory, which can serve as a good template / starting point for others to bundle their application:

  • README: A step by step instructions for generating the bundle
  • jhbuildrc: A custom jhbuildrc for building the bundle
  • AppRun: A script to run your GTK+ app from inside the bundled environment
  • PrepareAppDir.sh: A script to post-process the built stack including GTK+, it’s dependencies and your app
  • LibcWrapGenerator.vala: A program for generating libcwrap.h
  • modulesets/bundle.modules: A self-contained moduleset for building the stack
  • modulesets/patches/: A few downstream patches needed to build/run the bundle
  • triggers/: Some custom jhbuild post-installation triggers

People are welcome to copy this build/linux subdirectory into their project and try to build their own GTK+ based application bundles.

Different applications will have different requirements, you will certainly need to modify AppRun and bundle.modules.

 

Hand to hand combat with glibc

Most of my work this week has revolved around doing hand to hand combat with glibc. I do feel a bit resentful that gcc does not provide any -target-abi-version=glibc,2.7 sort of command line option, as this work would be much better suited to the actual compiler and linker, however with libcwrap.h and it’s generator, I’ve emerged victorious from this death match.

The technique employed to get the whole stack to depend on glibc 2.7 ABI involves using the .symver directive like so:

__asm__(".symver memcpy, memcpy@GLIBC_2.2.5");

The symbol memcpy() changed in libc’s 2.14 ABI to behave more like memmove(), however linking to the new symbol causes your application to require libc 2.14 ABI, which of course we don’t want. The above .symver directive ensures that you will link to the version of memcpy() which is provided by the glibc 2.2.5 ABI.

However the problem is more intense than just this, there are many symbols in libc with various versions and what we want is to be able to simply choose a target version. The LibcWrapGenerator.vala program takes care of this.

Running the generator on a given platform will call objdump on your system libc libraries and inspect which symbols exist, which of them can be safely accessed without any .symver directive, which symbols can be redirected to their most recent version before your arbitrary target ABI version and finally, of course, which ones simply are not available, for instance the fallocate symbol:

__asm__(".symver fallocate, fallocate@GLIBC_DONT_USE_THIS_VERSION_2.10");

The fallocate symbol was actually initially introduced in the 2.10 ABI, and simply cannot be used when targeting the glibc 2.7 ABI.

The nice thing about the above directive, is that it will cause an informative linker error when you try to link any program using fallocate() against glibc. Some of my patches work around these problems (for instance cairo uses longjump() in it’s test cases, so my patch simply disables these test cases when building for the bundle), in other cases this is totally transparent. For the case of fallocate above; libglib will try to use it if it’s available, but with the forced include of the generated “libcwrap.h” file, glib simply fails to find fallocate at configure time and happily functions without it.

This technique was borrowed from the now defunct autopackage project which used a bash script with awk to generate the header file in question, however this script stopped working for glibc runtimes >= 2.10. A newer port of that old script to vala was created by Jan Niklas Hasse in 2011 which was a good starting point, but still failed to generate the correct output. Today I put my (non-existent) vala skills to the test and now have a working version of this nifty header generator, salvaged from the autopackage wreckage.

 

More Testing !

I will of course appreciate any more testing of the new bundle, for any systems with the glibc 2.7 ABI or later. We did prove that the bundle works properly for a CentOS 6 system which is already a good start.

I would be interested however to know if other people can build the bundle following my README file in our build/linux directory, specifically I’m looking for cases where you have a very new glibc ABI, generate a new libcwrap.h, and let’s see if the bundling process still works, and also let’s see if we can run the bundled glade on very old systems, generated from very new systems.

 

That’s all, I think I’ve covered all the ground I intended to in this post… so please try this at home ! :)

And enjoy bundling your own GTK+ applications :)

 

Application Bundles for Glade

Tuesday, December 10th, 2013

This week started out with a search for some mechanism for building portable bundles for 64bit GNU/Linux systems. I looked at various bundler implementations, 0install, chakra, Alexander Larsson’s glick and glick2 to name a few of the implementations I found. Finally I found this gem called AppImageKit by Simon Peter (I refer to my own fork of his repository because it contains some patches I needed, however there is a pull request, which I think is the custom when doing things on github.com).

Before I continue, I should make clear what were my motivations and why I chose AppImageKit over any other existing options.

Why Bundled Applications ?

Bundled applications is a concept prominent in OSX and has been considered and reconsidered a lot by the general GNOME/GNU/Linux communities, I refer to Alex’s blog for some ideas on the concepts of bundling applications, you may also want to read this article by Lennart Poettering on the subject of Sandboxed Applications for GNOME (if you haven’t already seen his talk).

So first let me explain the motivations behind my hacking:

  • I wanted to make bleeding edge versions of Glade available to a wider user base. While many serious developers will naturally have a copy of GTK+ master from git.gnome.org built locally (as naturally you target next generation software for the apps you create), we still have many users who are just not into creating a relocated build system and building everything themselves.
  • Pretty much the same as the above point, I don’t want users to have to lag 3 years behind our development just because they decided to run a stable operating system (this should be a no brainer, really).
  • Finally, I wanted a method of distributing portable binaries of proprietary applications, without the fuss of wallowing around in the muck of creating rpms and debian packages for every different system out there, I want to just create one bundle which will run pretty much everywhere with X11 and 64bit linux. Yes, time to throw eggs and tomatoes at me in public. Just because I’ve contributed lots and lots of my own time to work on free software for ‘free as in beer’, and have been a proponent of Free Software for my own idealistic reasons (I just believe that code developed in public is held to a higher standard of quality), doesn’t mean that all the software I write is going to be free software.

Why did I choose AppImageKit ?

If you’re already familiar with the scene of Application Bundles out there, then you’ve already probably guessed why I made my choice.

In my search for application bundling mechanisms out there, I found a few options (as I mention in the beginning of this post). I quickly realized that most of the projects I found were either targeting a specific OS/distribution (i.e. chakra) or at least required the user to install some kind of mechanism to run the bundle (i.e. 0install). While I really do respect and admire the work done by Alexander and the advocacy by Lennart, pushing for a new concept of packaging in GNU/Linux systems, my requirements were simply different (and perhaps once sandboxed applications for GNOME actually exist, I will be able to switch to that new mechanism).

My requirement is, again, to be able to download and run an application on your 64bit GNU/Linux platform, I don’t care if it’s Fedora, GNOME OS, Debian, Arch Linux, Ubuntu, Chakra, I just wanted it to run.

How does it work ?

After reading the first paragraphs of the AppImageKit documentation I was sold. This is a simple and nifty idea of simply creating a compressed ISO image with an ELF binary header, it uses libz to decompress itself and fuse to mount itself into a temporary directory, and then it just runs a script, letting you modify your environment and launch your App from within the unpacked bundle environment (actually the AppRunScript.sh is my own addition to AppImageKit). It also uses libglib for some obscure reason, so the best practice is to build the bundling mechanism with the oldest version of libglib possible (as this will become a base runtime requirement for the user).

So basically the requirements are:

  • C runtime libraries
  • libfuse, with a fuse capable kernel
  • libz
  • libglib (as old as possible)

After that, the system requirements depend entirely on what you put in the bundle.

Please test my Glade bundle !

After 2 days of hacking I was finally able to get a GTK+ application running (Glade) with the Adwaita theme fully functional, and self contained. This involved much ldd and strace in the bundle environment, ensuring that the runtime doesn’t touch any system libraries or auxiliary data, the discovery of this nifty chrpath tool, and much grepping through various sources and figuring out how to properly relocate these modules with environment variables. I had to strangle Pango, until it finally yielded, after I thoroughly deformed pango’s face with this downstream patch. I also needed to rediscover hicolor-icon-theme’s location on fdo, since everything breaks down without an index.theme file in the right place (one of the various things jhbuild forgets to do properly).

You might be interested to peek at the README which is a thorough breakdown on how to create portable bundles of Glade in the future, the application startup script is also of particular interest as it shows what environment variables you really need to relocate GTK+ apps.

Finally, I was able to produce this bundle.

In addition to the AppImageKit system requirements listed above, we require:

  • X11 libraries
  • fontconfig
  • freetype

(Technically these could also be included in the bundle, it would require that we install fonts as well, and I just wanted to avoid that telling myself that most GNU/Linux distributions have this already).

The above bundle should be able to ‘just run’, just download it and run it without any special privileges… PLEASE !

We will greatly appreciate any testing on various 64bit GNU/Linux platforms, after some testing we will start to advertise downloadable bundles of bleeding edge Glade for any users of stable GNU/Linux distributions.

And of course, Enjoy !

Smart & Fast Addressbooks

Tuesday, December 3rd, 2013

This post is a sort of summary of the work we’ve done on the Addressbook for Evolution Data Server this year at Openismus. As I demonstrated in my previous post, Evolution’s addressbook is now riddled with rich locale sensitive features so I won’t cover the sorting features, you can refer to the other post for that.

 

Understanding Phone Numbers

This is another really nice feature which I admit has been driving me up the wall for the last year. I’ll try to sum it up one last time here so hopefully I don’t have to ever concern myself too much with the topic again ;-)

Before I get into it, I should start with describing the problem which we’ve addressed (I can’t say solved as I don’t believe there really is any ultimate solution for this problem).

So, this problem is particular to the implementation of a hand phone device, which implies the following conditions:

  • Users will enter any kind of data into the addressbook and call it a phone number. This most typically involves phone numbers entered as local numbers, sometimes it includes fully qualified international numbers for contacts who live abroad, and can also include text such as “ask jenny for her number next time”, as a rule, anything goes.
  • Addressbooks are typically a collection of vCards, this is a point of interest as using a standard format for contacts allows one to send vCards back and forth, which means that you cannot consider yourself in control of the data you store on your device. Instead a vCard can come from a synchronization of contacts from your laptop’s email client, or passed over bluetooth, the vCard can come from anywhere, containing any data it pleases and calling that a phone number.
  • When initiating an outbound call, the cellular modem firmware will send a string of numbers to the carrier, this will either succeed or fail. It’s tempting at this stage to consider and store the result of this operation, but unclear if the modem firmware will tell you the fully qualified number of the successful outbound call, and unreasonable to attempt a call just to determine such a thing.
  • When the firmware announces an incoming call, there will be a fully qualified E.164 phone number available (E.164 is a standard international phone number format).

Two obvious use cases now present a difficulty:

  • Let’s say the user starts entering a phone number somewhere in the UI, perhaps with the intent of initiating a phone call, or just with the intent of searching for a contact. We know that the user entered ‘just about anything’ as a phone number, we know that the vCard might have come from an external source (the same user might not have entered the phone number to begin with), and of course, we want to find the correct contact, and we want to find that contact right now.
  • Let’s imagine also, just for a moment, that your hand phone implementation might actually receive a phone call (not all of your users are so popular that they actually receive calls, but this is, after all, what your device is for ;-)). So now we have access to a fully qualified E.164 number for the incoming call, and an addressbook full of these vCards, which have, ya know, whatever the user decided to enter as a phone number. We of course want to find the single unique matching contact for that incoming phone number, and we want it even more righter and nower than in the other use case listed above (just in case, you know designers, maybe they want something crazy like displaying the full name of the contact who’s calling you instead of the phone number).

A common trick of the trade to handle these cases is to perform a simple heuristic which involves normalizing the number in your vCards (to strip out spaces and characters such as ‘-‘ and ‘.’) and then perform a simple suffix match against the incoming caller’s fully qualified phone number. This was admittedly a motivation behind our implementation of optimized suffix matching last year.

But let’s pretend that you’re not satisfied with this heuristic, you know you can do better, and you want to impress the crowd. Well now you can do just that ! assuming of course that you use Evolution’s addressbook in your device architecture (but of course you do, what else would you use ?).

 

Locale sensitive phone number interpretations

Again the International Components for Unicode (ICU) libraries come to the rescue, this time with the libphonenumber helper library on google code, which is now an optional dependency for Evolution Data Server (my painful experience compiling this heap of libphonenumber code begs me to make an unrelated comment: somebody school these guys in the ways of Automake, this CMake experience is just about the worst offence you can make to a downstream packager, or someone like me, who just needs to build it).

So what is a locale sensitive interpretation of a phone number ? Why do we need that ?

  • Some countries have different call prefixes for outgoing calls to leave the country. I believe this is ‘0’ in North America but it can vary. Even more confusing, take a country like Brazil with many competing cellular carriers, in the case of Brazil you have multiple valid calling prefixes to place a call outside of the country, each one with a different, competing rate.
  • Some locales, like in all locale sensitive affairs, have different preferences about how a phone number is composed, i.e. do we use a space or a dash or a decimal point as a separator ? do we interpret a trailing number preceded by a decimal as an extension code ? or as the final component of a local phone number ?
  • Some locales permit entirely different character sets to be used for numbers, and users might very well prefer to enter phone numbers in their native language script rather than entering standard numeric characters.

Using libphonenumber allows us to make the best possible guess of what the fully qualified E.164 number would be for an arbitrary string entered by the user in a given locale. It also provides us with useful information such as whether the phone number contained an extension code, whether the number had a leading 0 (special case for Italy) and whether the parsed phone number string had a country code. Actually libphonenumber will also make a guess at what the country code would be, but we’re not interested by that feature.

To leverage this feature in Evolution’s addressbook, one should make use of the following APIs:

  • The function e_phone_number_is_supported() can be called at runtime to determine if your installation of Evolution Data Server was compiled with support for phone numbers (as libphonenumber is only a soft dependency, this should be checked).
  • When creating a new addressbook, the ESourceBackendSummarySetup should be used to configure the E_CONTACT_TEL field into the addressbook ‘quick search’ configuration, and it should be configured with the special index E_BOOK_INDEX_PHONE so that interpreted phone numbers will be stored in SQLite for quick phone number matching.
  • Whether your addressbook is configured for quick phone number searches or not, so long as phone numbers are supported in your build then you have the capacity of leveraging the phone number matching semantics in the regular way with EBookQuery.

Three queries exist which can be used for phone number matching, at differing match strengths.

A typical query that you would use to search for, say equality at the national number strength level, would look like this:

EBookQuery *query = e_book_query_field_test (E_CONTACT_TEL,
                                             E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER,
                                             "user entered, or E.164 number");

With the above query one can use the regular APIs to fetch a contact list, filter a view, or filter a cursor.

 

Addressbooks on Crystal Meth

One thing that is really important of course in contacts databases, is speed, lots and lots of speed :)

While last year we’ve made considerable improvements, and already gained much speed for the addressbook, now contacts are inserted, deleted, modified and as specially fetched righter and nower than ever.

I should start by mentioning that in the last two weeks I’ve committed a complete rewrite of the old SQLite code which handles addressbooks, finally overcoming this bug, and turning what used to look like this (shudders) into something much more intelligible (sigh of relief). The net results in performance, can be observed here.

While I’d love to go over the vast improvements I’ve made in the code, and new features added, let’s just go through some highlights of the new benchmarks as I’ve been writing this post for a few hours already ;-)

First some facts about the addressbook configuration, since that makes a great difference in the meaning of the benchmark output.

  • E_CONTACT_GIVEN_NAME   – Forced fallback search routines (no index)
  • E_CONTACT_FAMILY_NAME  – Prefix Index, Suffix Index
  • E_CONTACT_EMAIL – Prefix Index, Suffix Index
  • E_CONTACT_TEL – Prefix Index, Suffix Index

The red line in the benchmarks is what is now in EDS master (although marked as “Experimental” in the charts). The comparisons are based on addressbooks opened in Direct Read Access mode, i.e. books are opened with e_book_client_connect_direct().

And now some of the highlights:

contact-saving

Total time to save all contacts, with varying numbers of contacts to add.

In the above chart we’re basically seeing the time we save by using prepared statements for batch inserts, instead of generating the same query again and again.

 

Time to fetch a contact by email prefix

Time to fetch a contact by email prefix

Prefix searches on contact fields which have multiple values (stored in a separate table) such as email addresses and phone numbers, now have good performance for addressbooks containing over 200,000 contacts (I was unable to test 409,600 contacts, as the benchmarks themselves require a lot of memory to keep a set of vCards and EContacts in memory).

 

filter-by-long-given-name-prefix

Fallback routine performed on the prefix searches of the given name

Here we see no noticeable change in the speed of fetching contacts with the fallback routine (i.e. no data to compare in SQLite columns, vCard parsing involved).

This is interesting only because in my new code base, we no longer fetch all contacts into memory and compare the list one by one, but instead install a function into SQLite to perform the fallback routine over the course of a full table scan. This is much more friendly to your memory consumption, and comes at no decrease in the performance of queries which hit the fallback routine.

 

Resident memory usage over the course of the benchmark progress

Resident memory usage over the course of the benchmark progress

Here we see the memory usage of the entire benchmark process including the evolution-addressbook-factory process over the course of the benchmarks (each dot from left to right is a measurement taken after each benchmark runs). Note that we have 200,000 contacts and vCards loaded into memory for the sake of the benchmarks to verify results for each speed test (hence the initial spikes at the beginning of the benchmark progress).

The noticeable drop in memory usage can be attributed to how the new backend is more friendly on system memory when performing fallback search routines.

Take a look at the full benchmarks including the csv file here. I should note that the ‘fetch-all-contacts.png’ benchmark indicated a bug where I was not detecting a special case (a wildcard query which should simply list all results), that has been fixed and the benchmark for it doesn’t apply with current Evolution Data Server master.

Anyway, it’s late and time for pizza :)

I hope you’ve all enjoyed the show !

 

Get your contacts sorted

Tuesday, October 29th, 2013

Hi all, I’ve been short on blog posts this summer and realize, after the fact, that I’ve not blogged once about the cursor API we’ve been cooking up at Openismus this year.

I’d like to present to you an interesting and unique new feature which will be available in the GNOME platform for 3.12, the cursor API to iterate over contacts in Evolution Data Server which has recently landed in git master.

Of course, the ability to sort a list of contacts and iterate over them efficiently can not be unique. I do think we have a solid API, and we support cursors in direct read access (bypassing D-Bus) but I’d really like to highlight some of the new internationalization features this API brings with it, allowing one to easily implement rich locale awareness in contact browsing applications.

Rich locale sensitive sorting with ICU

The cursor iterates over contacts in the order specified by the cursor’s configuration, and does so using a locale sensitive sort.

The ICU libraries provide a much richer set of collation options than the standard glibc’s implementation of strcoll() and friends. For the contacts database, specific ICU collation rules are preferred in order to get the sort order most appropriate for an addressbook (we normally prefer ‘phonebook’ order as opposed to a ‘dictionary’, ‘phonetic’ or other flavours of sort order).

Dynamic locale changes

So your device / desktop has a locale setting ?

Want your device / app / desktop to transition seamlessly from one locale to another ?

This won’t be a problem for your contact browsing interface. EDS’s addressbook will update it’s locale automatically when it’s notified of a system locale change (via the org.freedesktop.locale1 interface), automatically sorting the contacts in the new sort order and updating the active alphabet (cursor positions are necessarily lost and reset at locale change time).

Alphabet Introspection

Contact browsers, for your hand phone or your email application, are fun to navigate when you can jump to a given letter, or have some feedback showing what letter in your alphabet which you are currently browsing.

But what happens when your application must be usable in English, Greek, Japanese and Arabic ?

The ICU libraries again come to the rescue, providing us with tools we can use to display and navigate through characters and positions which are valid in the user’s active alphabet / script.

The new cursor API leverages this feature to make locale sensitive alphabetic browsing easy.

Diagram explaining how alphabetic positions relate to a contact list

Diagram explaining how alphabetic positions map onto a contact list

Since ICU provides us with functionality to determine what text should sort under which alphabetic position in the active alphabet, we are able to generate a sort key which will sort in between characters in a given alphabet.

This allows us to solve the problem of navigating to the position ‘E’ in the alphabet reliably and in the right place, as one must have knowledge of which variation of the letter ‘E’ (upper or lower case variations of e, é, è, ë, ê, etc…) sorts lowest in the active locale.

Documentation and Examples

This time around I wrote an obscene amount of documentation and a fully functional alphabetically sorted example contact browser.

Please enjoy the pretty diagrams and code snippets while writing your locale aware contact browsing applications :)

Below is the example contact browser first in en_US and then in ja_JP, unfortunately I did not take the time to add contact data from many locales (so there are no Japanese names listed there, instead all of the names in Latin script sort below the active Japanese alphabet).

Example contact browser en_US locale

Example contact browser en_US

Example contact browser in ja_JP locale

Example contact browser ja_JP

Not back from GUADEC yet …

Thursday, August 22nd, 2013

Good after noon gnomies ;-)

While many of you are already back home and settled in after GUADEC this year, I’m still scrambling around from hostel to hostel and city to city (or rather from town to town ?) in Europe.

So a few words about GUADEC are owed, as usual it was great to meet face to face with more people I usually only work with on IRC or communicate with by email, I met a few new faces this year (specifically Matthew Barnes and Milan Chra from the Evolution team) which was really nice.

Day One

On the first day of GUADEC I arrived at the hotel, August 1st, really looking forward to the first night where we all get together and have some beer and meet each other before the conference starts. Since I arrived early in the afternoon on the 1st, and nobody was around, I got to relax a bit and wait for the others to arrive (I’m thinking, YAY I guess I’m the first to arrive ;-) )… since I’m only speaking on Day 2 of the conference I should have plenty of time to dust off my presentation and prepare myself…

This fantasy of mine quickly disintegrated shortly after, when I realized that GUADEC had started without me, one entire day earlier than I had booked my flight & room for the dorms.

Happily, I ran into the Collabora Kids (Montreal Chapter) on the street while trying to use some wifi at the local grocery store (as wifi was not available at the dorms), and then was lead down to a bar where most of the GUADEC people could be found (Thanks for finding me !).

The Rest

So the rest of GUADEC went as expected, there were talks, I gave my talk about the new composite templates feature in GTK+, and on how this will help us to guide developers of GTK+ applications to create cleaner, more modular and more deterministic code using composite widgets. It was also my goal to point out how this has been sorely needed over the years which GTK+/Glade/libglade/GtkBuilder has evolved. My criticism towards myself is that I think I talk a bit slow, but hopefully I managed at least to get all my points across to the audience.

There was a meeting with the Evolution hackers, where I got to meet Matthew, Milan, Srini and others, Alberto chaired the meeting and this was definitely a very involved and productive meeting. We also took a picture of the Evolution team at the end which Alberto posted in his blog. Unfortunately we did not line up against the wall and appear to be monkeys which evolved into human beings and and then devolved into computer users with back problems… but the picture was still a great souvenir ;-)

There was of course plenty of face-to-face conversations, and plenty of late night beerings, Federico informed me of the tradition of SMASH (Single Malt Appreciation Society… Something beginning with ‘H’), I’m not sure if the H really stands for something or if you are supposed to be too drunk to care by the time you get to ‘H’, but we proudly kept up the tradition with a bottle I purchased in one of the airports on the way to Brno.

We even wrote code !

I know I know, we write code all year round, sitting in various parts of the globe, hanging out on IRC, so the last thing you would expect to do at GUADEC is write code, right ?

But near the end we did have a bit of a live hacking session on Glade with Juan Pablo, Kalev, Miguel and I (Kalev wrote the new support for GtkListBox which we reviewed, touched up, and pushed upstream, Miguel is new to writing code in C/GObject and we guided him with fixing an annoying bug in the signal editor).

All in all this was a pretty productive GUADEC and I’m very thankful to the GNOME foundation for sponsoring my trip to get here, thank you !

sponsored-badge-shadow

Composite templates lands in Vala

Wednesday, May 29th, 2013

This is another journal entry relating to the composite template saga. It’s been a little over a month now since Openismus was kind enough to sponsor my work on embedding of GtkBuilder xml directly into GtkWidget class data, allowing the automation of composite widget classes.

As of recently, composite template support has been merged into Vala master. This Vala improvement was brought to you by Luca Bruno, so if you find yourself saving a lot of time in your daily Vala development practice, please be encouraged to transfer at least part of these savings into Luca Bruno’s personal beer fund.

This is really exciting for me because this integration in high level languages is really what I’ve had in mind while working towards this goal. Of course, the developer experience would be greatly improved by IDE tools which use Glade’s core and additionally have knowledge of Vala syntax and provide some glue features to help automate these associations (having both Glade’s core and Vala’s parser and object structures in the same memory space would allow some awesome integration features)… at any rate, this is still a major milestone on the road to a bleeding edge developer experience.

From now on your UI developer experience should look like this

The new Vala constructs allowing one to integrate with GtkBuilder files and create composite widgets in Vala are documented here.

Using the Vala compiler in GNOME git master today, it is possible to compile this full demo.

And here is just a sample from that demo:

using Gtk;

[GtkTemplate (ui = "/org/foo/my/mywidget.ui")]
public class MyWidget : Box {
        public string text {
                get { return entry.text; }
                set { entry.text = value; }
        }

        [GtkChild]
        private Entry entry;

        public MyWidget (string text) {
                this.text = text;
        }

        [GtkCallback]
        private void button_clicked (Button button) {
                print ("The button was clicked with entry text: %s\n", entry.text);
        }

        [GtkCallback]
        private void entry_changed (Button button) {
                print ("The entry text changed: %s\n", entry.text);

                notify_property ("text");
        }
}

The above is basically what an example composite widget looks like in Vala code. Notice that there are three attributes which can be used to bind your class to a GtkBuilder xml file.

  • GtkTemplate: This attribute tells the compiler to bind your class with the specified .ui resource, it is up to you to ensure that the specified resource is compiled into your program (using GResource).
  • GtkChild: This indicates that the instance variable refers to an object defined in the template GtkBuilder xml
  • GtkCallback: This defines one of your object class methods as a callback suitable to be referred to in the GtkBuilder xml in <signal> statements. If a class method is defined as a [GtkCallback], then you can simply specify the method name as a handler in Glade’s signal editor and your method will be called in response to the signal emission defined in the GtkBuilder file.

A small challenge

Over recent years, we’ve been hearing various stories about what GNOME should ‘choose for an official binding’, my money has always been on Vala. This is not because I know how to write code in vala, it’s not because I feel comfortable with vala or anything of the sort.

My reasoning is simple:

  • Vala is a formalism that we control, since it is based on GObject it’s the most valid candidate to deliver the features which are specific to our platform in a comfortable way. Examples of this are the implied syntaxes which Vala provides to connect callbacks to GSignals, built-in language features for using asynchronous callbacks from GIO, and now, my favourite addition is of course the [GtkTemplate] syntax.
  • A programming language is a window into a feature set available in a given platform. It took me all of 2 days to learn Objective-C and after doing so it makes perfect sense that one would choose Objective-C with it’s platform specific feature set to integrate with the NextStep environment, because hey, that’s what it was created for (so why try to use basic C code, or Java to integrate with the NextStep environment if a customized Objective-C variant was explicitly created to make your developer experience richer for that platform ?)

So the challenge is this: Prove me wrong.

Do you have another favourite language binding for GNOME ?

If so, can you please extend your language’s formalism to define composite widget classes ?

Whether this is really a challenge or not is yet to be seen, I’m really curious ;-)

 

DoggFooding in Glade

Friday, April 26th, 2013

I’ve been meaning to write a short post showing what we’ve been able to do with Glade since we introduced composite widget templates in GTK+, the post will be as brief as possible since I’m preoccupied with other things but here’s a run over of what’s changed in the Dogg Food release.

Basically, after finally landing the composite template machinery (thanks to Openismus for giving me the time to do that), I couldn’t resist going the extra mile in Glade, over the weekends and such, to leverage the great new features and do some redesign work in Glade itself.

So please enjoy ! or don’t and yell very loudly about how you miss the old editor design, and make suggestions :)

Glade Preferences Dialog

Preferences Dialog Before

Preferences Dialog Before

Preferences Dialog After

Preferences Dialog After

 

 

 

 

 

 

 

The old preferences dialog was a sort of lazy combo box, now that we have composite templates and create the interface using GtkBuilder, it was pretty easy to add the treeview and create a nicer interface for adding customized catalog paths.

Also there are some new features and configurations in the dialog, since the new Dogg Food release we now have an Autosave feature, and we optionally save your old file.ui to a file.ui~ backup every time you save. There are also some configurations on what kind of warnings to show when saving your glade file (since it can be annoying if you already know there are deprecated widgets, or unrecognized widgets and have the dialog popup every time you save).

Glade Project Properties

Project Properties Dialog Before

Project Properties Dialog After

 

 

 

 

 

 

 

Refactoring out the project properties dialog into a separate source file, and implementing the UI with Glade makes the GladeProject code more readable, also the UI benefits again, notice the not so clear “Execute” button has been moved to be a secondary dialog button (with a tooltip explaining what it does).

Also the new project attributes have been added to allow one to set the project’s translation domain or Composite Template toplevel widget.

Now that’s just the beginning, let’s walk through the new custom editors.

Button Editor

GtkButton Editor After

GtkButton Editor After

GtkButton Editor Before

GtkButton Editor Before

 

 

 

 

 

 

 

 

 

Here’s where the fun starts, while we did have some custom editors before, they all had to be hand written, now I’ve added some base classes making it much easier to design the customized property editors with Glade.

First thing to notice is we have these check button property editors for some boolean properties which we can place wherever in the customized property editor layout (checkbuttons previously didnt make any sense in a layout where one always expects to see the title label on the left, and the property control on the right, in a table or grid layout).

Entry Editor Before

GtkEntry Editor Before (top portion)

GtkEntry Editor Before (top portion)

GtkEntry Editor Before (bottom portion)

GtkEntry Editor Before (bottom portion)

 

 

 

 

 

 

 

 

 

 

 

 

Entry Editor After

GtkEntry Editor After (top portion)

GtkEntry Editor After (top portion)

GtkEntry Editor After (bottom portion)

GtkEntry Editor After (bottom portion)

 

 

 

 

 

 

 

 

 

 

 

 

All around better layout I think, also we save space by playing tricks with the tooltip-text / tooltip-markup properties for the icons. While in reality GTK+ has separate properties, we just add a “Use Markup” check to the tooltip editor and use that to decide whether to edit the normal tooltip text property, or the tooltip markup property.

Image Editor

GtkImage Editor Before

GtkImage Editor Before

GtkImage Editor After

GtkImage Editor After

 

 

 

 

 

 

 

 

 

Here we economize on space a bit by putting the GtkMisc alignment and padding details down at the bottom, also we group the “use-fallback” property with the icon name setting, since the fallback property can only apply to images that are set by icon name.

Label Editor

GtkLabel Editor Before

GtkLabel Editor Before

GtkLabel Editor After

GtkLabel Editor After

 

 

 

 

 

 

 

 

 

 

 

 

 

Like the GtkImage Editor, we’ve grouped the GtkMisc properties together near the bottom. We also have generally better grouping all around of properties, hopefully this will help the user find what they are looking for more quickly. Another interesting thing is that the mnemonic widget editor is insensitive if “use-underline” is FALSE, when “use-underline” becomes selected, the mnemonic widget property can be set directly to the right of the “use-underline” property.

Widget Editor / Common Tab

Last but not least (of what we’ve done so far) is a completely new custom editor for the “Common” tab (perhaps we can do away with the “Common” tab altogether… use expanders where we currently have bold heading labels, now that we do it all with GtkBuilder script, sky is the limit really)

GtkWidget Editor Before (top portion)

GtkWidget Editor Before (top portion)

GtkWidget Editor Before (bottom portion)

GtkWidget Editor Before (bottom portion)

 

 

 

 

 

 

 

 

 

 

Widget Editor After

GtkWidget Editor After

GtkWidget Editor After

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Here again we play some tricks with the tooltip, so that we don’t have two separate property entries for “tooltip-text” and “tooltip-markup” but instead a simple switch deciding whether to set the tooltip as markup or not. The little “Custom” check button in there makes the tooltip editors insensitive and instead sets the “has-tooltip” property to TRUE, so that you can hook into the “query-tooltip” signal to create a custom tooltip window.

Now while these are just the first iterations of the new editor designs and layouts, the really great news is that we can now use Glade to design the layouts of Glade’s widget editors, so you can expect (and even request ;-)) better designs in the future. Also, we’re open to ideas, if you have any great ideas on how to make widget property editing more fun, more obvious, more usable… please share them with us in bugzilla or on our mailing list.

 

Extra amendment: Fitting images into blog post side by side has been a delicate exercise, it looks different in the editor, different at blogs.gnome.org, and again different on planet.gnome.org, just goes to show that I make for a terrible poster boy, not to mention I don’t post quite that often… anyway… hope the formatting of this post is endurable at least, it’s best viewed at blogs.gnome.org I think.

Announcing Composite Widget Templates

Tuesday, April 9th, 2013

Hello fellow hackers, today we bring you a new feature which I believe can very much improve the GTK+/GNOME developer story.

This is a feature I’ve been planning for a long time (I originally blogged about it 3 years ago) so I’m very excited about it having finally landed in GTK+, it’s my hope and ambition that this feature will help shape the future of user interface programming with GTK+.

Before I continue, I have to thank Juan Pablo Ugarte for keeping the dream alive and talking about this at the last GUADEC. Also recognition must be given to Openismus GmbH for sponsoring my full time work on this for the last few weeks, the time to completely focus on this task would not have been afforded me without them.

Unfortunately this post will be a little terse, the one I had planned which is a bit more relaxed and has some comic relief will not be ready on time. So, at the risk of being taken seriously, let’s continue with a brief overview of the APIs introduced to GTK+ and the actions taken.

What are Composite Widget Templates ?

Composite Widget Templates are an association of GtkWidget class data with GtkBuilder xml, which is to say that the xml which defines a composite widget is now a part of the definition of a widget class or type.

This feature automates the creation of composite widgets without the need for directly accessing the GtkBuilder APIs and comes with a few features that help to bind a GtkWidget with it’s GtkBuilder xml.

As of yesterday, 23 composite widget classes in GTK+, from simple classes such as GtkFontButton or GtkVolumeButton to more complex widget classes such as GtkFileChooserDefault and GtkPrintUnixDialog have all been ported to remove all manual user interface creation code, in favour of GtkBuilder xml.

So, how can I use it ?

There are three or four new APIs added to GtkWidget which play on the class data, currently they will only be available in C, but if you have a little imagination, you can see how this can be very useful in higher level languages, by extending the syntax and adding some keywords (hint: I have vala in mind as a top candidate).

Before I go into the API details here, I would like to point out a complete working example which I created today while writing this post. To give it a try, you need of course GTK+ master from today (or yesterday). For those who are interested I suggest you download that small tarball, and build it with one simple ‘make’ command.

First, lets start with an example of how to bind your template to your widget class:

static void
my_widget_class_init (MyWidgetClass *klass)
{
  GtkWidgetClass *widget_class = GTK_WIDGET_CLASS (klass);

  /* Setup the template GtkBuilder xml for this class
   */
  gtk_widget_class_set_template_from_resource (widget_class, "/org/foo/my/mywidget.ui");
}

static void
my_widget_init (MyWidget *widget)
{
  /* Initialize the template for this instance */
  gtk_widget_init_template (GTK_WIDGET (widget));
}

So, to bind some GtkBuilder XML to a widget class, we need to call two functions:

  • gtk_widget_class_set_template_from_resource() binds some GtkBuilder XML to the class data
  • gtk_widget_init_template() initializes the template for a given instance, this is currently needed for the base C apis, but both could certainly be automated in a highlevel language.

Next, we have a function which creates an implicit relationship between some instance variables and some objects defined in the GtkBuilder XML:

struct _MyWidgetPrivate
{
  /* This is the entry defined in the GtkBuilder xml */
  GtkWidget *entry;
};

static void
my_widget_class_init (MyWidgetClass *klass)
{
  GtkWidgetClass *widget_class = GTK_WIDGET_CLASS (klass);
  GObjectClass *gobject_class = G_OBJECT_CLASS (klass);

  /* After having called gtk_widget_class_set_template_from_resource(), we can
   * define the relationship of the private entry and the entry defined in the xml.
   */
  gtk_widget_class_bind_child (widget_class, MyWidgetPrivate, entry);

  g_type_class_add_private (gobject_class, sizeof (MyWidgetPrivate));
}

In the above code, we’ve defined a relationship between the MyWidgetPrivate pointer named ‘entry’ and the object in the GtkBuilder XML of the same name ‘entry’. The entry will be available for access on the subclassed GtkWidget instance private data at any time after gtk_widget_init_template() was called, until the given widget is disposed (at which time the pointer will become NULL). GTK+ takes care of memory managing such automated pointers, so it is ensured to exist for the lifetime of your instances.

Again, with highlevel bindings in mind, this could be implemented as some syntactic sugar in the actual declaration of the instance variable.

Finally there is one more point of interest in the API which is Callbacks. Functions in your widget class code can be specified as Callbacks which serve as endpoints for any signal connections defined in the GtkBuilder XML.

/* A callback handling a "clicked" event from a button defined in the GtkBuilder XML */
static void
my_widget_button_clicked (MyWidget  *widget,
                          GtkButton *button)
{
  g_print ("The button was clicked with entry text: %s\n",
           gtk_entry_get_text (GTK_ENTRY (widget->priv->entry)));
}

static void
my_widget_class_init (MyWidgetClass *klass)
{
  GtkWidgetClass *widget_class = GTK_WIDGET_CLASS (klass);

  /* After having called gtk_widget_class_set_template_from_resource(), we can
   * declare callback ports that this widget class exposes, to bind with <signal>
   * connections defined in the GtkBuilder xml
   */
  gtk_widget_class_bind_callback (widget_class, my_widget_button_clicked);
}

Note that all signal connections defined in composite templates have the composite widget instance as user data by default.

In the above example code, my_widget_button_clicked() callback was declared with the assumption that the <signal> connection defined in the template was declared as ‘swapped’. Swapped signal connections are connections where the user data of the callback is returned first instead of the emitter. I think that this should be the default for composite widget callbacks as it blends in more naturally with normal class methods (where the class instance is always the first parameter).

This detail might not apply directly to higher level languages, which could achieve the above by adding some syntactic sugar in the declaration of a Callback method. Perhaps the instance will be implied as the ‘self’ variable.

I have also included some additional API to allow bindings to specify the GtkBuilderConnectFunc which should be used to make signal connections for a given widget class. I hope that bindings authors will contact me if they need any additional support in the GTK+ api to implement this.

Can I now use Glade to define my Composite Widget Templates ?

Of course silly ! That’s the whole point right ?

You’ll need Glade master from today as well, however I should be rolling a development snapshot with full support for this later this week as well.

All of GTK+’s composite widget classes have been recreated using Glade. Here is a screenshot of a GTK+ composite widget being edited in Glade:

Glade editing the GtkFileChooserDefault

Glade editing the GtkFileChooserDefault

Glade is still in it’s early stages supporting this, so there is hardly any features added here. I hope to work towards a brighter future where Glade can understand a multitude of composite widget templates as components of a single project, which will open the doors for some really nice and useful features.

Conclusion

All in all I have a lot to say about this work, but I’ll cut this blog post short for now, however I may be posting followups in the near future.

I’m very satisfied with this work, and I hope you will enjoy creating user interfaces as composite widget classes.

Where did my memory go ? – A detective story

Tuesday, February 12th, 2013

Here, is yet another follow up post on EDS memory consumption. For the last few days I’ve been tracking where memory is spent in EDS and our benchmarking tools, and it was a very interesting experience.

And I’m not just saying that ! it was very trying and it’s still a bit of an unsolved mystery to me (so please feel free to step in with your theories on the unsolved parts !).

It all started when Michael asked me to explain the funny spikes in the memory usage graph presented in the previous post. The first thing I did was to produce a more “bumpy” graph by disabling the slice allocator, yielding what is in some ways a more accurate account of actual memory usage:

Memory usage measured for 12,800 contacts with G_SLICE=always-malloc

Memory usage measured for 12,800 contacts with G_SLICE=always-malloc

Interestingly, I say “in some ways” above because; one of the elements that we have to consider is memory fragmentation; memory management is generally more optimal and less fragmented when the slice allocator is active.

What we are looking at above is a left to right graph of overall memory usage; measured after each and every operation that we run on the addressbook. Each “dot” can be associated to one of the various latency tests that we run for each and every build of EDS (indicated in the legend).

First of all let’s demystify the “curious humps” which occur mostly to the “Custom Light” (light blue) benchmarks but are also noticeable in other benchmarks. These “humps” occur for four dots at a time, particularly when performing suffix searches on contact fields that are not stored in the summary SQLite tables for quick searches.

This phenomenon is partly attributable to the fact that all contacts in the addressbook need to be individually examined (and the vcards individually parsed) when the given contact field is not stored in the SQLite tables individually (or what we refer to in EDS terms as “the summary”). I’m not really very concerned by these “spikes”; obviously the memory is reclaimed later on, however it is curious that this happens specifically for suffix matching and not for prefix matching (presumably lot’s of extra string duplications and normalizations are needed for the case insensitive suffix matching routines).

Now that that’s out of the way, it leads us to some of the…

More interesting parts

I was at first not satisfied with only this explanation, sure, it kindof explains the “funny humps” in the benchmark progress but… by taking a closer look at what else is actually happening… I needed a better explanation.

The portions of the presented memory usage graphs that interest me more are the memory growth observable over the course of the first four dots, as well as the curious memory growth that also occurs at the very end of the benchmarks.

So what is happening in these stages ?

First of all, it’s positive news to know that the number of automatically generated vcards used for testing are already in memory before the benchmarks start at all, in the above graph that represents 12,800 vcards all in memory before the first benchmark is measured. And then…

  1. The addressbook is initialized and created, so at the point of measuring the very first dot, we have 12,800 vcards in memory and an initialized EBookClient on the client side and an addressbook counterpart (SQLite database created and active SQLite connection) in the server side memory
  2. Next, at the second dot we’ve created 12,800 EContact objects in memory… the 12,800 EContacts and 12,800 vcard strings remain in memory throughout the benchmark progress. This second dot is about 45MB higher on the scale than the first dot, so it’s pretty safe to say that 12,800 EContact objects cost roughly 45MB of resident memory which will not be reclaimed for the duration of the benchmark progress.
  3. The third dot is measured directly after adding all the contacts to the addressbook, here we start to see some divergence in memory usage; notice that this costs roughly 25MB extra for EBookClient based benchmarks, but only about 5MB for EBook based benchmarks. Being a bit naive, I overlooked this detail at the beginning of the investigation… one of the notable differences in the EBook apis is that it was lacking in batch commands. So the major difference here is that EBook tests add contacts one by one over D-Bus, while the EBookClient tests add contacts in batches of 3200 contacts at a time.
  4. This fourth dot, is after fetching all contacts at once from the addressbook. Here is where I became seriously alarmed. For normal clients, this shows an approximate 30MB growth in memory consumption. So where did my memory go ? A simple case of amnesia ?! Note though, that the Direct Read Access (red) benchmark hardly increases in memory for a fetch of 12,800 contacts, good show.

Naturally, feeling embarrassed about the consequences of the evil fourth dot… I frantically started my search for memory leaks… first I blamed the obscure nature of C++ code and it’s attempts to hide memory management behind smart pointers…I tried to pin it as a memory leak in the actual benchmarking code (after all, I did just lose 30MB of memory… it must have gone somewhere… right ?)… but after some tracing around, I found that those returned contacts, stored by smart pointers or not, were properly finalized and freed, leaving me with this uncomfortable mystery still on my hands.

While most of my memory leak hunt revolved around explaining the 30MB memory overhead incurred from dot 3 to dot 4, I should mention that the last memory jump was also suspicious. This last memory jump (which seems to vary between a 10MB to 25MB increase depending on the benchmark type) is incurred by deleting all contacts in the addressbook. So how about that ? I’ve just deleted all the contacts, and now I’m using MORE memory than before ?

The following day…

… I ran the benchmarks in loops, for some I’ll share below because this is how I eventually solved the mystery case, I also ran the benchmarks (server and client) under valgrind, ran some various test cases with the server and test cases running under valgrind. But the alleged memory leak was not to be tracked. Some testing of the benchmarks running in a loop seemed to indicate that there was some memory growth over time, not very much so, but enough to make me believe there must be some leak and be determined to find out.

Finally, today…

… I let my laptop chug along and loop the benchmarks (at least some of them) with a huge 12,800 contact count (that takes time), so let’s share those enlightening results here:

Memory usage while benchmarking 12,800 contacts in the first iteration

Memory usage benchmarking 12,800 contacts in the second iteration

Memory usage benchmarking 12,800 contacts in the second iteration

Memory usage benchmarking 12,800 contacts in the third iteration

Memory usage benchmarking 12,800 contacts in the third iteration

These results would be better viewed from left to right instead of one on top of the other, but you get the idea. Just consider that the last dot in the first chart happens directly before the first dot in the following chart, and so on.

So, after viewing this data… we can see that in the second and third graph, memory we presumed to be lost, is eventually returned to the system (in other words, it was indeed only a case of temporary amnesia, and not a more severe degrading case of alzheimer’s)… This is very reassuring, numerous runs with valgrind also show no real evidence of memory leakage, which is also reassuring evidence that our EDS is leak free.

But, that still doesn’t really explain…

Where is that memory actually going ?

At this point I can only give you my best guess, but all of the clues seem to point towards D-Bus traffic:

  • At the second “dot” where contacts are added to the addressbook, EBook APIs adding only a single contact at a time seems to cost much much less than using EBookClient apis and adding the contacts in batches of 3200 contacts at a time.
  • At the third “dot” where a brute “fetch all contacts” call is made to the addressbook, we can see a huge increase in memory consumption all except for when using Direct Read Access mode. So when fetching a list of 12,800 contacts not using D-Bus, we don’t suffer from memory loss.
  • In the last suspicious “dot”, where we delete all contacts from the addresssbook at once, all benchmark types seem to suffer significant memory loss. In this case the client is sending a list of 12,800 contact UIDs over D-Bus to the addressbook (in Direct Read Access as well, since deleting contacts is a write operation).

My best guess ? this is all due to zero-copy IPC transfers implemented by D-Bus.

In other words (if you’ve read up to this point you probably don’t need any explanation), instead of the sender writing chunks of data to a socket, and the receiver reading bytes from a socket; the sender is owning some shared memory which is accessed directly by the receiver.

This shared memory is probably managed by the D-Bus daemon itself, so it would make sense that the daemon not release the shared memory straight away but instead reserve some head room in the case that further transfers might reuse that memory.

So how come the fourth dot where a batch of 12,800 vcards are passed to the client, is not reused by the last dot where all contacts are deleted ? … Because, when contacts are fetched the shared memory owner would have to be the sender, which is the addressbook server. However when contacts are deleted, it is the EBookClient user process which sends a list of 12,800 UIDs, in this case the owner of the shared memory should be the other, client process.

I’ll probably need to pursue some extra verifications to be sure, but this best guess is very compelling to me at this time.

In conclusion, this was a really interesting exercise, which I don’t hope to repeat very often… but I did learn a few things and it did put some things into perspective. First and foremost; measuring memory usage, when compared to just tracking and plugging leaks, is quite another story… a lot more tricky and probably not an exact science.

If you’ve got this far, I hope you’ve enjoyed this detective story… I did enjoy it.

Amendments

It’s probably bad form but I’ll just add this here, my theory is obviously false. As I’ve been informed (already) that D-Bus does not implement any such zero-copy mechanisms with shared memory… so there is still a huge memory fluxuation, definitely related to D-Bus usage, which I can’t readily explain.