Just another GNOMEr

Roadmap to 0.7

While I was at the desktop summit, I decided to come up with a roadmap so we all had something to work to for the 0.7 unstable release which we are hoping to do soon. The roadmap is on live.gnome.org here:

http://live.gnome.org/Tracker/Roadmap

As you can see, it is progressing nicely.

Config

The configuration system in Tracker has always consisted of one TrackerConfig inheriting from a GObject and used to load/save applying the GKeyFile API. The problem we found here, is that we really want configurations to be more fine grained to specific binaries. Some of the options (like log verbosity) would apply to ALL binaries that use the config otherwise. So now we have TrackerConfigFile as a base class with tracker-object-keyfile.[ch] to do some utility functions for us in libtracker-common and all binaries that want their own TrackerConfig with object properties now inheirt from TrackerConfigFile. This is quite nice because it reduces the code duplication we had and now we have a nice set of separate config files in $HOME/.config/tracker/.

With 0.6. we also had this concept of “modules” which would be for each type of data we wanted to track. We had “files”, “applications”, “email”, and some others… These modules also have a configuration pertaining to how to index their data. Things like globs for including and ignoring certain files. There are also options to make sure data isn’t indexed too regularly (which was needed for some content that was constantly updating). All of this is in the process of being revised and merged with the TrackerConfig machinery. This mostly applies to the “files” module though. The module config and module code (which was a complex GModule implementation) is all going to be simplified now that we have separate binaries for mining each data we are interested in.

Album Art

This was quite a mess before. We had code in different places for this. Over the past week or so I have cleaned this up too. Now we do all album art downloads and extraction from the tracker-extract binary (called from mp3 and gstreamer extractors when they see media with such content). Before we would request thumbnails for the new art in tracker-extract, but due to the unstable nature of tracker-extract (based on dynamically loading modules using 3rd party APIs we can’t guarantee the stability of) we were always at risk of failing to queue new thumbnail requests to the thumbnail daemon if we had a crash. We only send thumbnail requests AFTER all indexing has been completed, if we don’t do this, we suffer with severe performance problems. Now all thumbnail requests are done from one place, the tracker-miner-fs and the albumart functions are no longer spread across libtracker-common and tracker-extract. They are just in tracker-extract.

Volume Support

Over the past few days Carlos re-added volume support to Tracker so now using a simple query, you can find out if your data on that MMC you just inserted or removed is available.

So, to get a list of ALL data objects and their availability (which is true or false based on if the media is mounted or not) you can use:

$ tracker-sparql -q "SELECT ?do ?av WHERE {
                            ?do a nie:DataObject ;
                            tracker:available ?av }"

You can also get a list of all data objects which are NOT available. To make things faster, we have not included “available” for EVERY item, only where items are available. This makes updating the tables a lot faster. So when looking for files which are not mounted, the query becomes a bit more complex:

$ tracker-sparql -q "SELECT ?do WHERE {
                            ?do a nie:DataObject .
                     OPTIONAL {
                            ?do tracker:available ?av } .
                     FILTER (! BOUND(?av)) }"

Of course the most common use case is, tell me files which are available, which can be done with:

$ tracker-sparql -q "SELECT ?do WHERE {
                            ?do a nie:DataObject ;
                            tracker:available true }"

We are still fine tuning the volume work to be faster but things are coming along swimmingly!

Desktop Summit

Wanted to say thank you to everyone at the desktop summit this year. It was superb and it was good to see everyone again!

So it became quite obvious to me at this years desktop summit in Gran Canaria that no one really knows what is going on with regards to Lanedo and my involvement in projects. This is primarily because I haven’t been blogging enough. I have decided to change this.

After speaking to various people (Bastien, Lucas, etc) I was surprised to hear some of the questions about Lanedo. I thought my initial blog covered it. But clearly not.

Lanedo & Imendio

In December 2008, Micke spoke to us all in Imendio and said that he was going to shutdown the company. Of course this came as a huge surprise to everyone given our success over the years and the economic climate was not the reason for his decision. The reason was stress. If I really think back I can see how Micke was trying to change things internally to alleviate this by of-loading some of his responsibility to others. This happened probably for a year or more. In the end, I think it was just too much. Towards the end of Imendio, you could tell how stressed Micke was by his demeanor. Now-a-days, he is much happier and everyone can see the change.

Richard decided to not continue with Imendio too. As such Tim and I (who were effectively internally managing projects at Imendio) decided to start a new company if everyone (except Micke and Richard) wanted to continue. The consensus was that they did, so in January 2009, Lanedo GmbH was formed in Hamburg. We took on some of Imendio’s contracts and now we are continuing the work on our own steam.

Tracker

This year Tracker was in the spotlight somewhat. As a project it was grown considerably in the last 12 months. In the early part of last year, Carlos and I started working on it full time. More and more people got involved like Jürg Billeter, Philip Van Hoof, Ivan Frade and Mikael Ottela. These are the core developers. We refactorred a lot of it to produce the 0.6.9x releases. Jamie has been providing feedback about direction and ideas and doing one of the most important features – the SQLite module we use for Full Text Search (FTS).

About 3-6 months ago, Jürg, Philip and Ivan started looking into the 0.7 work and at the moment Jurg is leading the development there while I maintain bug fixes for the 0.6 branch. Our roles in the project are all quite well defined (I would say at least) and it is a really fun project to work on with some really brilliant people contributing. Right now, this is how it looks:

I handle the File system monitoring, crawling and database connection management. I also do the 0.6.9x releases and have been doing project management in coordination with Urho Konttori.

Carlos maintains the indexing of the data, the extensions (or modules) which know what to do with the data we extract.

Philip works on the thumbnailing and has a really good appetite for creating specifications and working with new technologies to provide ideas about how to improve areas.

Ivan is our ontology guru not to mention he added the GLib unit tests to Tracker which is a huge benefit.

Jürg has been working on completely refactoring the databases and the higher level API that sits on top of them (libtracker-data). Jürg is also leading development the 0.7 (master) branch right now.

Mikael is our extractor expert. Mikael has been improving constantly the MP3/GStreamer/JPEG/etc extractors to get better performance for each release.

ALL of us do general project maintenance it should be added, so we all contribute to each other’s areas. These are also just some of the more noteable areas which we are each involved in. It is a large project and there are a lot of things not mentioned here.

So right now Tracker is looking really good and it is an exciting project to be involved in, especially with Zeitgeist being interested in using it and other components in BOTH desktops too.

I plan to blog much more about features we add, crap we remove, etc.

Lanedo has only been running since January and we have been lucky enough to be able to sponsor the conferences we usually attend this year.

So far, these include the Desktop summit and Linuxtag and we are also looking into sponsoring FSCONS later in the year. For LinuxTag we are sponsoring by sending Tim and Sven and by donating to the cause.

This year as usual, Sven will be propping up the GNOME booth for us, so if you are in that vicinity, don’t forget to come by and say hello!

Baby Coming!

Sue is about to have our baby (expected date is the 18th of March), she really can’t wait for it to be born now and neither can I! Right now I am just trying to get as much sleep as possible in preparation
We don’t yet know if it is a boy or a girl so there is an added excitement after waiting 9 months not knowing. Sue thinks it is a boy, I think it is a girl.

Tracker Release

Yesterday I released Tracker 0.6.91, which follows the recent 0.6.90 release that we did after 12 months of solid development on the project. I say we, there is quite a huge team working on this project now, including Carlos Garnacho, Ivan Frade, Jürg Billeter, Philip Van Hoof, Mikael Ottela, Urho Konttori and many more. We have a preliminary roadmap (as mentioned here) for Tracker  too. This recent release and possibly one more will be the last before 0.7 which will include Jürg’s vstore branch (which we have been working on in parallel for months now). We also had a discussion about the current architecture of the project and decided to change some of the roles around regarding what the indexer and daemon currently do to make things more efficient. With this all in mind, I am expecting some seriously good fun on this project in the next 3 months.

Where did we come from?

Lanedo GmbH has taken over the majority of the business that Imendio AB has been running over the past 5 years as the founders have decided to step back for personal reasons.

Lanedo GmbH is still employing the same people that have GTK+ and Maemo expertise with the exception of the founders of Imendio AB.

I haven’t done much on Gossip in a while. Been rather preoccupied with Tracker recently.

Today I managed to add a feature which uses the text under the mouse pointer to search with Google, Wikipedia and Youtube. Right now those are just hard coded, but I had in mind, to make it configurable.

Web Services

Recently I set up a new blog for GTK+ to document releases and for people to blog about cool new things in GTK+. So far, there is just Andreas and myself on the blog. So if anyone else wants to be added just let me know your blog email account and I will set it up. If you have been missing the GTK+ project news, don’t forget, it can be see at http://planet.gnome.org/news/, it is not shown on the regular http://planet.gnome.org/.

Andreas has been doing a great job chasing up some final issues on gtk.org, mainly with regards to the language bindings. Andreas has now removed all bindings that have no support for 2.6 and above and all bindings which are supported by GNOME are illustrated nicely too. So, if your language binding isn’t listed there and you know it is supported, please let us know on the gtk-devel list so we can rectify the pages!

Yep, I am off to New Zealand for my vacation from February 1st to the 29th with Sue.

I started working on the new design for gtk.org last April and tonight the new pages were uploaded and we now have a spiffing new look which I am really happy with.

I want to thank Andreas Nilsson for helping out with the design and Tim Janik for help with the final touches in getting the pages installed, not to mention everyone who commented.