Carlos Garnacho

It happens sneakily, but there’s more things going on in the Tracker front than the occasional fallout. Yesterday 2.2.0-alpha1 was released, containing some notable changes.

On and off during the last year, I’ve been working on a massive rework of the SPARQL parser. The current parser was fairly solid, but hard to extend for some of the syntax in the SPARQL 1.1 spec. After multiple attempts and failures at implementing property paths, I convinced myself this was the way forward.

The main difference is that the previous parser was more of a serializer to SQL, just minimal state was preserved across the operation. The new parser does construct an expression tree so that nodes may be shuffled/reevaluated. This allows some sweet things:

• Property paths are a nice resource to write more idiomatic SPARQL, most property path operators are within reach now. There’s currently support for sequence paths:

  # Get all files in my homedir
  SELECT ?elem {
    ?elem nfo:belongsToContainer/nie:url 'file:///home/carlos'
  }
  

  
  And inverse paths:
  

  # Get all files in my homedir by inverting
  # the child to container relation
  SELECT ?elem {
    ?homedir nie:url 'file:///home/carlos' ;
             ^nfo:belongsToContainer ?elem
  }
  

  There’s harder ones like + and * that will require recursive selects, and there’s the negation (!) operator which is not possible to implement yet.

• We now have prepared statements! A TrackerSparqlStatement object was introduced, capable of holding a query with parameters which can be set/replaced prior to execution.

  conn = tracker_sparql_connection_get (NULL, NULL);
  stmt = tracker_sparql_connection_query_statement (conn,
                                                    "SELECT ?u { ?u fts:match ~term }",
                                                    NULL, NULL);
  
  tracker_sparql_statement_bind_string (stmt, "term", search_term);
  cursor = tracker_sparql_statement_execute (stmt, NULL, NULL);
  

  This is a long sought protection for injections. The object is cacheable and can service multiple cursors asynchronously, so it will also be an improvement for frequent queries.

• More concise SQL is generated at places, which brings slight improvements on SQLite query planning.

This also got the ideas churning towards future plans, the trend being a generic triple store as much sparql1.1 capable as possible. There’s also some ideas about better data isolation for Flatpak and sandboxes in general (seeing the currently supported approach didn’t catch on). Those will eventually happen in this or following cycles, but I’ll reserve that for other blog post.

An eye was kept on memory usage too (mostly unrealized ideas from the performance hackfest earlier this year), tracker-store has been made to automatically shutdown when unneeded (ideally most of the time, since it just takes care of updates and the unruly apps that use the bus connection), and tracker-miner-fs took over the functionality of tracker-miner-apps. That’s 2 processes less in your default session.

In general, we’re on the way to an exciting release, and there’s more to come!

Last evening I came back from the GNOME performance hackfest happening in Cambridge. There was plenty of activity, clear skies, and pub evenings. Here’s some incomplete and unordered items, just the ones I could do/remember/witness/talk/overhear:

• Xwayland 1.20 seems to be a big battery saver. Christian Kellner noticed that X11 Firefox playing Youtube could take his laptop to >20W consumption, traced to fairly intensive GPU activity. One of the first things we did was trying master, which dropped power draw to 8-9W. We presumed this was due to the implementation of the Present extension.
• I was looking into dropping the gnome-shell usage of AtspiEventListener for the OSK, It is really taxing on CPU usage (even if the events we want are a minuscule subset, gnome-shell will forever get all that D-Bus traffic, and a11y is massively verbose), plus it slowly but steadily leaks memory.

  For the other remaining path I started looking into at least being able to deinitialize it. The leak deserves investigation, but I thought my time could be better invested on other things than learning yet another codebase.

• Jonas Ådahl and Christian Hergert worked towards having Mutter dump detailed per-frame information, and Sysprof able to visualize it. This is quite exciting as all classic options just let us know where do we spend time overall, but doesn’t let us know whether we missed the frame mark, nor why precisely would that be. Update: I’ve been pointed out that Eric Anholt also worked on GPU perf events in mesa/vc4, so this info could also be visualized through sysprof
• Peter Robinson and Marco Trevisan run into some unexpected trouble when booting GNOME in an ARM board with no input devices whatsoever. I helped a bit with debugging and ideas, Marco did some patches to neatly handle this situation.
• Hans de Goede did some nice progress towards having the GDM session consume as little as possible while switched away from it.
• Some patch review went on, Jonas/Marco/me spent some time looking at a screen very close and discussing the mipmapping optimizations from Daniel Van Vugt.
• I worked towards fixing the reported artifact from my patches to aggressively cache paint volumes. These are basically one-off cases where individual ClutterActors break the invariants that would make caching possible.
• Christian Kellner picked up my idea of performing pointer picking purely on the CPU side when the stage purely consists of 2D actors, instead of using the usual GL approach of “repaint in distinctive colors, read pixel to perform hit detection” which is certainly necessary for 3D, but quite a big roundtrip for 2D.
• Alberto Ruiz and Richard Hughes talked about how to improve gnome-software memory usage in the background.
• Alberto and me briefly dabbled with the idea of having specific search provider API that were more tied to Tracker, in order to ease the many context switches triggered by overview search.
• On the train ride back, I unstashed and continued work on a WIP tracker-miners patch to have tracker-extract able to shutdown on inactivity. One less daemon to have usually running.

Overall, it was a nice and productive event. IMO having people with good knowledge both deep in the stack and wide in GNOME was determining, I hope we can repeat this feat again soon!

There’s been a rumor that I was working on improving gnome-shell on-screen keyboard, what’s been up here? Let me show you!

The design has been based on the mockups at https://wiki.gnome.org/Design/OS/ScreenKeyboard, here’s how it looks in English (mind you, hasn’t gone through theming wizards):

The keymaps get generated from CLDR (see here), which helped boost the number of supported scripts (c.f. caribou), some visual examples:

As you can see there’s still a few ugly ones, the layouts aren’t as uniform as one might expect, these issues will be resolved over time.

The additional supported scripts don’t mean much without a way to send those fancy chars/strings to the client. We traditionally were just able to send forged keyboard events, which means we were restricted to keycodes that had a representation in the current keymap. On X11 we are kind of stuck with that, but we can do better on Wayland, this work relies on a simplified version of the text input protocol that I’m doing the last proofreading before proposing as v3 (the branches currently use a private copy). Using an specific protocol allows for sending UTF8 strings independently of the keymap, very convenient too for text completion.

But there are keymaps where CLDR doesn’t dare going, prominent examples are Chinese or Japanese. For those, I’m looking into properly leveraging IBus so pinyin-like input methods work by feeding the results into the suggestions box:

The suggestion box even kind of works with the typing booster ibus IM. But you have to explicitly activate it, there is room for improvement here in the future.

And there is of course still bad stuff and todo items. Some languages like Korean neither have a layout, nor input methods that accept latin input, so they are badly handled (read: not at all). It would also be nice to support shape-based input.

Other missing things from the mockups are the special numeric and emoji keymaps, there’s some unpushed work towards supporting those, but I had to draw the line somewhere!

The work has been pushed in mutter, gtk+ and gnome-shell branches, which I hope will get timely polished and merged this cycle 🙂

Tracker is one of these pieces of software that get no special praise when things work, but you wake up to personal insults on bugzilla when they don’t, today is one of those days.

Several distros have been eager to push SQLite 3.20.0 still hot from the oven to their users, apparently ignoring the API and ABI incompatibilities that are described in the changelog. These do hit Tracker, and are only made visible at runtime.

Furthermore, there is further undocumented ABI breakage that makes FTS5 modules generated from pre/post 3.20.0 SQLite code backward and forward incompatible with the other versions. Tracker used to ship a copy of the FTS5 module, but this situation is not tenable anymore.

The solution then? Making it mandatory that SQLite >= 3.20.0 must have FTS5 builtin. The just released Tracker 1.12.3 and 1.99.3 will error hard if that is not the case.

I’ve just sent this mail to distributor-list:

Hi all,

Sqlite 3.20.0 broke API/ABI for Tracker purposes. The change described in point 3 at http://sqlite.org/releaselog/3_20_0.html is not only backwards incompatible, but also brings backwards and forwards incompatibilities with standalone FTS5 modules like Tracker ships [1], all of those are only visible at runtime [2].

FTS5 modules generated from SQLite < 3.20.0 won't work with >= 3.20.0, and the other way around. Since it's not tenable to ship multiple FTS5 module copies for pre/post 3.20.0, Tracker shall now make it a hard requirement that SQLite is compiled with builtin FTS5 (--enable-fts5) if SQLite >= 3.20.0 is found. The current Tracker FTS5 module is kept for older SQLite versions.

This change applies to Tracker >=1.12.3 and >=1.99.3. I don't know if any distro pushed SQLite 3.20.0 in combination with a Tracker that is older than that, but it will be just as broken, those would require additional patches from the tracker-1.12 branch besides the described change.

Please handle this promptly, wherever there's sqlite 3.20.0 without builtin FTS5 and/or tracker <= 1.12.2/1.99.2, there's unhappy Tracker users.

Cheers,
  Carlos

[1] Generated as described in
http://sqlite.org/fts5.html#building_a_loadable_extension
[2] https://bugzilla.gnome.org/show_bug.cgi?id=785883

So, if you see errors about “TrackerTokenizer” something, please contact your distro packagers. I’ll close further incoming bugs as NOTGNOME.

There’s a few plans in the boiler for Tracker:

Splitting core from miners

Tracker is usually deemed a “metadata indexer”, although that’s just half the truth. Even though Tracker could be essentially considered that in its very early days, it made a bold move back in 0.7.x to using Sparql as the interface to store and retrieve this metadata, where both the indexers and the applications using this metadata talk the same language.

So in reality, the storage and query language are useful by themselves. As per the plans, you’ll now have to distinguish between:
– Tracker the RDF store, and
– Tracker miners, the infrastructure and binaries to translate a number of entities (be it files or whatever) into Sparql, using the first acceptation for storage

Making standalone Sparql endpoints possible

At the time of moving to Sparql, Tracker was conceived as a global store of deeply interconnected data, Nepomuk seemed the most obvious choice to represent the data for indexing purposes and client isolation was basically left to 3rd parties. However times change, sandboxing is very present, and Tracker’s global store don’t help with it.

Based on initial work from Philip Van Hoof, quite some shuffling has been going on in wip/carlosg/domain-ontologies to make multiple Sparql endpoints (“database connections” in non-fancy speak) possible. This will allow applications to use private sparql endpoints, and possibly using other ontologies (“schemas” in non-fancy speak) than Nepomuk. The benefits are twofold, this will be a lot friendlier to sandboxing, and also increments the versatility of the Tracker core.

Switching to Semantic versioning

This change was suggested some time ago by Philip Van Hoof, and the idea has been growing in me. Tracker has usually had a longstanding backwards compatibility promises, not just in terms of API, but also in terms of data compatibility. However, bumping the version as per the GNOME schedule results in multiple stable versions being used out there, with the patch backporting and bug management overhead that it implies.

In reality, what we want to tell people (and usually do!) is “use latest stable Tracker”, often the stuff is already fixed there, and there’s no reason why you would want to stick to a stable series that will receive limited improvements. I do hope that semantic versioning conveys Tracker’s “later is better” stance, with some optimism I see it standing on 2.y.z for a long time, and maybe even 2.0.z for tracker core.

But versions are like age, just a number :). Tracker 2.x services and client side API will foreseeably be backwards compatible with 1.x from the reader standpoint. The Tracker core could be made parallel installable with 1.0, but I wouldn’t even bother with the high level infrastructure, 2.x will just be a better 1.x.

But this doesn’t mean we jump the GNOME unstable cycle ship. IMHO, it’s still worth following to let newly introduced code bake in, it just won’t result in gratuitous version bumps if API changes/additions are not in sight.

Code and API cleanup

In the more active past, Tracker had a lot code accretion while trying to get the buy-in, this included multiple miners, extensions and tools. But it was never the goal of Tracker to be the alpha and omega of indexing in itself, rather to have applications update and blend the data for their own consumption. Fast forward a few years and the results are mixed, Tracker got an amount of success, although apps almost exclusively rely on data produced by Tracker’s own miners, while most of these extensions are bitrotting since much of the activity and manpower went away.

Sam Thursfield started doing some nice cleanups of maemo/meego specific code (yes, we still had that) and making Tracker use Meson (which indirectly started tiptoeing some of that bitrot code). Several of these these extensions implementing Tracker support shall just go to the attic and should be done proper, it will at least be the case with nautilus thanks to Alexandru Pandelea’s GSOC work :).

But the version bump and code split is too good of an opportunity to miss it :). Some deprecated/old API will also go away, probably none of which you’ve ever used, there will be some porting documentation anyway.

Under public domain

There’s been lately lots of fuzz around Tracker as a security risk, as the de-facto maintainer of Tracker I feel obliged to comment. I’ll comment purely on Tracker bits, I will not comment on other topics that OTOH were not as debated but are similarly affected, like thumbnailing, previewing, autodownloading, or the state of maintenance of gstreamer-plugins-bad.

First of all, I’m glad to tell that Tracker now sandboxes its extractors, so its only point of exposure to exploits is now much more constrained, leaving very little room for malicious code to do anything harmful. This fix has been backported to 1.10 and 1.8, and new tarballs rolled, everyone rejoice.

Now, the original post raising the dust storm certainly achieved its dramatic effect, despite Tracker not doing anything insecure besides calling a closed well known set of 3rd party libraries (which after all are most often installed from the same trusted sources that Tracker comes from), it’s been on the “security” spotlight across several bugs/MLs/sites with different levels of accuracy, I’ll publicly comment on some of these assertions I’ve seen in the last days.

This is a design flaw in Tracker!

Tracker has always performed metadata extraction in a separate process for stability reasons, which means we already count on this process possibly crashing and burning away.

Tracker was indeed optimistic at the possible reasons why that might happen, but precisely thanks to Tracker design it’s been a breeze to isolate the involved parts. A ~200 lines change hardly counts as a redesign.

All of tracker daemons are inherently insecure!, or its funnier cousin Tracker leaks all collected information to the outside world!

This security concern has only raised because of using 3rd party parsers (well, in the case of the GStreamer vulnerability in question, decoders, why a parsing facility like GstDiscoverer triggers decoding is another question worth asking), and this parsing of content happens in exactly one place in your common setup: tracker-extract.

Let’s dissect a bit Tracker daemons’ functionality:

• tracker-store: It is the manager of your user Tracker database, it connects to the session bus and gets readwrite access to a database in ~/.cache. Also does notification of changes in the database through the user bus.
• tracker-miner-fs: It’s the process watching for changes in filesystem, and filling in the basic information that can be extracted from shared-mime-info sniffing (which usually involves matching some bytes inside the file, little conditionals involved), struct dirent and struct stat.
• tracker-extract: Guilty as charged! It receives the notification of changes, and is basically a loop that picks the next unprocessed file, runs it through 3rd party parsers, sends a series of insert clauses over dbus, and picks the next file. Wash, rinse, repeat.
• tracker-miner-applications: A very simplified version of tracker-miner-fs that just parses the keyfiles in various .desktop file locations.
• tracker-miner-rss: This might be another potential candidate, as it parses “arbitrary” content through libgrss. However, it must be configured by the user, it otherwise has no RSS feeds to read from. I’ll take the possibility of hijacking famous blogs and news sites to hack through tracker-miner-rss as remote enough to fix it after a breathe.

So, taking aside per-parser specifics, tracker consists of one database stored under 0600 permissions, information being added to it through requests in the dbus session, and being read by apps from a readonly handle created by libtracker-sparql, the read and write channels can be independently isolated.

If you are really terrified by your user information being stored inside your homedir, or can’t sleep thinking of your session bus as a dark alley, you certainly want to run all your applications in a sandbox, they won’t be able to poke on org.freedesktop.Tracker1.Store or sniff on ~/.cache that way.

But again, there is nothing that makes Tracker as a whole inherently insecure, at least not more than the average session bus service, or the average application storing data in your homedir. Everything that could be distrusted is down to specific parsers, and that is anything but inherent in Tracker.

Tracker-extract runs plugins and is thus unsafe!

No, tracker-extract has a modular design, but is not extensible itself. It reads a closed set of modules implemented by Tracker from a folder that should be in /usr/lib/tracker-1.0 if your setup is right. The API of these modules is private and subject to change. If anything manages to add or modify modules there, you’ve got way worse concerns.

Now, one of these extractor modules uses GStreamer, which to my knowledge is still the go-to library if you want anything multimedia on linux, and it happens to open an arbitrary list of plugins itself, that is beyond Tracker control or extent.

It should be written in rust!

What do we gain from that? As said, tracker-extract is in essence a very simple loop, all the scary stuff is handled by external libraries that will still be implemented in “unsafe languages”, rust is just as useful as gift paper to wrap this.

Extraction should be further isolated into another process!

There are good reasons not to do that. Having two separate processes running completely interlocked tasks (one process can’t do anything until the other is finished) is pretty much a worst case for scheduling, context switching, performance and battery life at once.

Furthermore, such tertiary service would need exactly the same whitelisted syscalls and exactly the same number of ways out of the process. So I think I won’t attract the “Tracker is heavy/slow” zealots for this time… There is a throwaway process, and it is tracker-extract.

The silver linings

Tracker is already more secure, now lets silence the remaining noise. Quite certainly one area of improvement is Flatpak integration, so sandboxed applications can launch isolated Tracker instances that run under the same sandboxed environment, and extracted data is only visible within the sandbox.

This is achievable with current Tracker design, however the “Tracker as a service” approach sounds excessive with this status quo, tracker needs to adapt to being usable as a local store, and it needs to go through being more of a generic SPARQL endpoint before.

But this is just adapting to the new times, Flatpak is relatively young and Tracker is slow moving, so they haven’t met yet. But there is a pretty clear roadmap, and we’ll get there.

So this is finally happening. The result of much work all through the stack from several dedicated individuals (You know who you are!) started lining up during the past few months and now is hitting master. Early in the cycle I blogged about stylus support being merged, based on the first version of the tablet protocols. Now I have the pleasure to declare GTK+ tablet support on Wayland feature complete.

As predicted in the earlier post, a second version of the protocol came through, bringing pad support for clients. What is a pad? It’s the set of buttons and other (often) tactile sensors that tablets have around the stylus-sensitive area. These devices are rather uncanny from an input management perspective: unlike mice/keyboard, those buttons/sensors don’t have an associated action that has been chiseled on the key or through decades of user interaction paradigms, they are rather meant to be user-mappable. Also, despite buttons being buttons and sensors being essentially one-dimensional axes, all resemblance with a mouse/stylus is purely coincidental, focus management is more similar to keyboards (actually, the same), and is not directly related to the stylus.

In GNOME, this action-mapping has been traditionally done in gnome-settings-daemon. X11 clients have been usually completely unaware of pad events, partly because of the oddities pointed out above. So g-s-d kept a passive grab on pad buttons and would translate those into keycombos. This has many shortcomings though, keycombos are far from standard, accelerators are translatable, … For Wayland, we have the opportunity to fix all these shortcomings, and make pads a first class citizen.

So the right thing to do here, if we want to univocally map actions to pad features, is delegating the action mapping to the client. The other side of the coin is providing proper feedback about the actions. In GNOME we have session-wide OSDs to display the pad mapping, and the wayland protocol has been tailored to cover an usecase like this, so applications can directly participate in filling in this info, this is how the end result looks:

* Note to supervillains: Actions are stubs

There’s also changes scheduled (I know I’m late, who doesn’t like pressure!) to gnome-control-center to centralize the management of styli for all known/plugged tablets, following the very nice mockups from Jimmac.

How is this exposed in GTK+?

gtk+ master introduced GtkPadController, it’s a GdkEventController subclass that will manage pad events, from one pad specifically, or for all at once. The controller takes a GActionGroup and a set of pad action entries, each defining an action name to activate, a simple example:

controller = gtk_pad_controller_new (GTK_WINDOW (app_window),
                                     G_ACTION_GROUP (app_window),
                                     pad_device /* May be NULL for all */);
gtk_pad_controller_set_action (controller,
                               GTK_PAD_ACTION_BUTTON,
                               1, /* Second button, they're 0-indexed */
                               -1, /* All modes */
                               _("Frobnicate file"),
                               "app.frobnicate-file");

And pressing that button on that pad will trigger the related action from the GActionGroup. The given label is what you end up seeing in the OSD. I expect this object to be eventually made useful on other platforms than Wayland (X11 is the first objective), although only the event-to-action mapping, and not entirely exempt of platform-specific issues.

This is going to be messy! why does every app need configuring? Why not a global action map?

The Wayland protocol aims to be intemporal, the trouble would only begin at defining a good enough initial set of global actions, and still not all actions might make sense for every app. This allows every application to implement pad actions, no matter how high or low level they are. Configurability is also optional, although the first apps that will surely come to your head will implement some, they already did for styli and other input features.

Is this for me?

Depends :), if you see your app might be involved in artists or designers’ workflow, it well could. Providing a sensible minimal set of actions to perform, even if hardcoded (think those you’d like shortcuts for, and were away from the keyboard), could help lots in integrating this previously neglected device, to the point that it might feel widely useful besides the niche drawing application case. By using GAction underneath, there’s also little left for you to test, you just need to ensure the action performs the intended effect when activated.

As for excuses go for blowing the lint off my blog, this is a pretty good one :). Wayland tablet support is something that got really close to being merged in 3.20, but the timing didn’t pan out in the end, wayland-protocols 1.3 included a tablet v1 protocol that went unused, till now. Now that we’re in 3.22, those bits are at last being merged. First came gtk+, which you can see working in these videos:

This improved tablet support brings some goodies (also on X11), there is a new GdkDeviceTool object that can used to track specific physical tools and provides the necessary info to do so in a persistent way (eg. across runs/reboots).

A note to all GTK+ app/widget developers

TL;DR: If you’re keeping track of motion events somewhere in your app, you want to check gdk_event_get_device(), see the bullet points below and check the best strategy for you.

One particularity in wayland tablet management is that whenever you have a tool in proximity to a tablet you’ll be driving a standalone onscreen pointer cursor, as opposed to the default behavior in X11 consisting of driving the same pointer than your mouse does. This difference is not moot, it doesn’t take too much thinking to realize that the user of a drawing tablet has a spare hand which could be using the mouse, so there’s in practical terms two pointers to react to. And there’s also setups where more than one drawing tablet is likely (the simplest one being a laptop with an integrated tablet plus a external one).

GTK+ is itself quite well prepared for these situations, as are all its widgets, it can’t account for widgets implementations elsewhere and signal callbacks in apps though.

But I’m not doing a drawing app, should I care?

Yes you should. Although tablets are mostly thought out for drawing, design, etc., ideally they should be able to drive the whole desktop without oddities, just as your mouse does. This can only drive to a pleasant experience with your app regardless of the input devices of choice.

If there’s somewhere in your app where you’re tracking pointer events over time (drawing is obviously an example, but so is rubberband selection, any kind of drag-to-move, etc…), there’s a chance that it could get confused by events with different gdk_event_get_device()s coming to the widget in question. There’s several things you might do:

• First and foremost, look into GtkGesture and its several subclasses, it’s handled transparently there, and there’s a good chance that one of those (or a group) will fit your case
• Maybe it’s sufficient a “first to come gets served” basis for the specific purposes of your operation? You can use gtk_device_grab_add/remove with that device, or check gdk_event_get_device() yourself on all events.
• Think it’d be cool to handle the simultaneous pointers and want to go the extra mile? Again, track gdk_event_get_device(), and keep per-device state.

…And there’s more to come

There’s big plans for mutter in this cycle so tablet support will not be immediately merged, but be assured it will happen in due time. Gears are also grinding in order to have a version 2 of the tablet protocol, it will mostly focus on bridging pads (the sets of buttons and sliders along the sides of some tablets) to wayland clients, something which I expect to be an outstanding improvement compared to X11, so… stay tuned!

In free software some fashions never change, and some are particularly hard to overcome. Today I’ll talk about the “Tracker makes $ANYTHING slow” adage, lately gnome-music being on the spotlight here. I’m glad that I could personally clear this up to some individuals on the hackfests/conferences I’ve been around lately.

But convincing is a never ending labor, there’s still confused people around the internets, and disdainful looks don’t work as well over there. The next best thing I could do is fixing things myself to make Tracker look less like the bad guy. So, from the “can’t someone else do it” department, here’s some commits to improve the situation. The astute reader might notice that there is nothing about tracker in these changes.

There’s of course more to it, AFAICT other minor performance hits are caused by:

• grilo emitting one signal per media item found, which is somewhat bad on huge lists
• icon view performance generally sucking, which makes scrolling not as smooth in the Albums view while covers are loading
• After all that, well sure, Tracker queries can be marginally optimized.

This will eventually hit master and packages, until then, do me a favor an point to this post anyone still saying how Tracker made gnome-music slow.

Developer experience hackfest

Kind of on topic with this, I attended a few weeks ago to the Developer experience hackfest. Besides trying to peg round pieces into square holes, after some talking with how much of a steep barrier was Sparql as a prerequisite for accessing Tracker data, I started there on a simpler query API that abstracted all of these gritty details. Code is just shaping up there, but I expect it to cover the most common usecases. I must thank Red Hat and Collabora for enabling me to go there, all the people there, and particularly Philip for being such a great host.

Oh, and also attended Fosdem and Devconf, even talked on the last one about the input plans going on in GNOME, busy days!

With 3.14 almost out of the door, it seems like a good opportunity to blow the cobwebs of this blog and highlight some shiny new features I was involved in during this cycle:

Gesture support in GTK+

It’s finally happening :), GTK+ 3.14 brings in infrastructure to handle gestures. Maybe the word “gesture” is automatically associated to “multitouch”, but actually this infrastructure is meant to deal with all kinds of pointer/touch input, and as such is used fairly intensively now within GTK+ itself, so even mouse users will be unknowingly using this.

These gesture objects are of course readily available for applications too. Individually, these are quite simple in essence, but can be easily stitched together to compound higher-level behavior. So far, eog and evince (and by extension gnome-documents) have bitten the bullet and now handle some of the gestures you’d expect on touchscreens there, the API documentation and HowDoI are available for anyone wanting to follow.

Gesture support in gnome-shell

Just to feed the cabal claiming that gnome-shell is designed for tablets, I also managed this cycle to add gesture infrastructure in mutter, so it is able to pre-process touch events before applications do, these events are then handled through ClutterGestureActions, or “rejected” and eventually handled by the application. This has been put to use in gnome-shell, making some actions readily available through multitouch.

Edge swipes

Showing the overview

Switching workspaces

Switching apps in a workspace

Note: Recorded with the help of this, bug #732367 is yet another 3.16 todo item…

Freebase support in libgdata

This one feature has certainly went underpublicited, and I found myself with little time to make use of it :(, but I nonetheless find that very interesting things can be done with this. Freebase is a community-maintained knowledge base (currently backed by Google), as seen on its homepage it is extremely wide in topics (some better covered than others), and has a very well defined ontology, think of it as a companion to Tracker on the web.

There are dedicated methods for the most usual ways to query data (search, lookup on topic…), but additionally Freebase offers a powerful MQL query method. MQL is very analogous to SPARQL, with the main difference that it’s based on JSON. All together allows for querying in very creative ways from very various data, a few examples being:

• The mandatory “query for movie/album info” example, actually these topics are the best covered.
• Fetching stock images for movies/cities/landmarks/directors/…, you name it.
• Looking up monuments close to a geolocation.
• Getting direct links to Wikipedia, in your language.

Looking forward for 3.15

It is almost time to celebrate, but I evidently won’t sit twiddling my thumbs :), a few items I’d like to tackle on the next cycle are:

• During the 3.14 cycle I got briefly started on adding optional gesture integration to GtkStack and a new “tabs” widget, now it sounds like a good time to resume. I would also like to make gestures used integrally for event handling in GTK+ (we’re already just a few widgets away from that goal)
• There’s a few gaps still left to solve on handling touchpad gestures, which I’d like to get closed ASAP, at least for touchpads handling >2 fingers on X11.
• Improving gnome on Wayland. I merely got started this cycle adding DnD/clipboards support to GTK+ and bringing touchscreen behavior on mutter more or less on par to X11’s. There’s a few input details that need shuffling so they’re done in the same place on X11/wayland (pointer cursor visibility, device mapping…), and I hope the timing to be right to bring in a sort of tablet support (libinput and wayland protocol details have been shaping up despite my times on, most times off help, thanks Peter, Lyude, Jason et al!), I will be putting my hacking efforts wherever it’s necessary to make this happen.
• WebKitGTK+ could be definitely made friendlier on touchscreens, additionally to the DOM touch event support it already does, it would be great to handle touch scroll/pinch/zoom as you can see in other pure GTK+ apps now.