Carlos Garnacho

“Colored net”by Chris Vees (priorité maison) is licensed under CC BY-NC-ND 2.0

Some years ago I was asked to come up with some support for sandboxed apps wrt indexed data. This drummed up into Tracker 2.0 and domain ontologies, allowing those sandboxed apps to keep their own private data and collection of Tracker services to populate it.

Fast forward to today and… this is still largely unused, Tracker-using flatpak applications still whitelist org.freedesktop.Tracker, and are thus allowed to read and change content there. Despite I’ve been told it’s been mostly lack of time… I cannot blame them, domain ontologies offer the perfect isolation at the cost of the perfect duplication. It may do the job, but is far from optimal.

So I got asked again “we have a credible story for sandboxed tracker?”. One way or another, seems we don’t, back to the drawing board.

Somehow, the web world seems to share some problems with our case, and seems to handle it with some degree of success. Let’s have a look at some excerpts of the Sparql 1.1 recommendation (emphasis mine):

RDF is often used to represent, among other things, personal information, social networks, metadata about digital artifacts, as well as to provide a means of integration over disparate sources of information.

A Graph Store is a mutable container of RDF graphs managed by a single service. […] named graphs can be added to or deleted from a Graph Store. […] a Graph Store can keep local copies of RDF graphs defined elsewhere […] independently of the original graph.

The execution of a SERVICE pattern may fail due to several reasons: the remote service may be down, the service IRI may not be dereferenceable, or the endpoint may return an error to the query. […] Queries may explicitly allow failed SERVICE requests with the use of the SILENT keyword. […] (SERVICE pattern) results are returned to the federated query processor and are combined with results from the rest of the query.

So according to Sparql 1.1, we have multiple “Graph Stores” that manage multiple RDF graphs. They may federate queries to other endpoints with disparate RDF formats and whose availability may vary. This remote data is transparent, and may be used directly or processed for local storage.

Let’s look back at Tracker, we have a single Graph Store, which really is not that good at graphs. Responsibility of keeping that data updated is spread across multiple services, and ownership of that data is equally scattered.

It snapped me, if we transpose those same concepts from the web to the network of local services that your session is, we can use those same mechanisms to cut a number of drawbacks short:

• Ownership is clear: If a service wants to store data, it would get its own Graph Store instead of modifying “the one”. Unless explicitly supported, Graph Stores cannot be updated from the outside.
• So is lifetime: There’s been debate about whether data indexed “in Tracker” is permanent data or a cache. Everyone would get to decide their best fit, unaffected by everyone else’s decisions. The data from tracker-miners would totally be a cache BTW :).
• Increases trustability: If Graph Stores cannot be tampered with externally, you can trust their content to represent the best effort of their only producer, instead of the minimum common denominator of all services updating “the Graph Store”.
• Gives a mechanism for data isolation: Graph Stores may choose limiting the number of graphs seen on queries federated from other services.
• Is sandboxing friendly: From inside a sandbox, you may get limited access to the other endpoints you see, or to the graphs offered. Updates are also limited by nature.
• But works the same without a sandbox. It also has some benefits, like reducing data duplication, and make for smaller databases.

Domain ontologies from Tracker 2.0 also handle some of those differently, but very very roughly. So the first thing to do to get to that RDF nirvana was muscling up that Sparql support in Tracker, and so I did! I already had some “how could it be possible to do…” plans in my head to tackle most of those, but unfortunately they require changes to the internal storage format.

As it seems the time to do one (FTR, storage format has been “unchanged” since 0.15) I couldn’t just do the bare minimum work, it seemed too much of a good opportunity to miss, instead of maybe making future changes for leftover Sparql 1.1 syntax support.

Things ended up escalating into https://gitlab.gnome.org/GNOME/tracker/commits/wip/carlosg/sparql1.1, where It can be said that Tracker supports 100% of the Sparql 1.1 syntax. No buts, maybe bugs.

Some notable additions are:

• Graphs are fully supported there, along with all graph management syntax.
• Support for query federation through SERVICE {}
• Data dumping through DESCRIBE and CONSTRUCT query forms.
• Data loading through LOAD update form.
• The pesky negated property path operator.
• Support for rdf:langString and rdf:List
• All missing builtin functions

This is working well, and is almost drop-in (One’s got to mind the graph semantics), making it material for Gnome 3.34 starts to sound realistic.

As Sparql 1.1 is a recommendation finished in 2013, and no other newer versions seem to be in the works, I think it can be said Tracker is reaching maturity. Only HTTP Graph Store Protocol (because why not) remains the big-ish item to reasonably tell we implement all 11 documents. Note that Tracker’s bet for RDF and Sparql started at a time when 1.0 was the current document and 1.1 just an early draft.

And sandboxing support? You might guess already the features it’ll draw from. It’s coming along, actually using Tracker as described above will go a bit deeper than the required query language syntax, more on that when I have the relevant pieces in place. I just thought I’d stop a moment to announce this huge milestone :).

Some personal highlights:

Emoji OSK

The reworked OSK was featured a couple of cycles ago, but a notable thing that was still missing from the design reference was emoji input.

No more, sitting in a branch as of yet:

This UI feeds from the same emoji list than GtkEmojiChooser, and applies the same categorization/grouping, all the additional variants to an emoji are available as a popover. There’s also a (less catchy) keypad UI in place, ultimately hooked to applications through the GtkInputPurpose.

I do expect this to be in place for 3.32 for the Wayland session.

X11 vs Wayland

Ever since the wayland work started on mutter, there’s been ideas and talks about how mutter “core” should become detached of X11 code. It has been a long and slow process, every design decision has been directed towards this goal, we leaped forward on 2017 GSOC, and eg. Georges sums up some of his own recent work in this area.

For me it started with a “Hey, I think we are not that far off” comment in #gnome-shell earlier this cycle. Famous last words. After rewriting several, many, seemingly unrelated subsystems, and shuffling things here and there, and there we are to a point where gnome-shell might run with --no-x11 set. A little push more and we will be able to launch mutter as a pure wayland compositor that just spawns Xwayland on demand.

What’s after that? It’s certainly an important milestone but by no means we are done here. Also, gnome-settings-daemon consists for the most part X11 clients, which spoils the fun by requiring Xwayland very early in a real session, guess what’s next!

At the moment about 80% of the patches have been merged. I cannot assure at this point will all be in place for 3.32, but 3.34 most surely. But here’s a small yet extreme proof of work:

Performance

It’s been nice to see some of the performance improvements I did last cycle being finally merged. Some notable ones, like that one that stopped triggering full surface redraws on every surface invalidation. Also managed to get some blocking operations out of the main loop, which should fix many of the seemingly random stalls some people were seeing.

Those are already in 3.31.x, with many other nice fixes in this area from Georges, Daniel Van Vugt et al.

Fosdem

As a minor note, I will be attending Fosdem and the GTK+ Hackfest happening right after. Feel free to say hi or find Wally, whatever comes first.

It happens sneakily, but there’s more things going on in the Tracker front than the occasional fallout. Yesterday 2.2.0-alpha1 was released, containing some notable changes.

On and off during the last year, I’ve been working on a massive rework of the SPARQL parser. The current parser was fairly solid, but hard to extend for some of the syntax in the SPARQL 1.1 spec. After multiple attempts and failures at implementing property paths, I convinced myself this was the way forward.

The main difference is that the previous parser was more of a serializer to SQL, just minimal state was preserved across the operation. The new parser does construct an expression tree so that nodes may be shuffled/reevaluated. This allows some sweet things:

• Property paths are a nice resource to write more idiomatic SPARQL, most property path operators are within reach now. There’s currently support for sequence paths:

  # Get all files in my homedir
  SELECT ?elem {
    ?elem nfo:belongsToContainer/nie:url 'file:///home/carlos'
  }
  

  
  And inverse paths:
  

  # Get all files in my homedir by inverting
  # the child to container relation
  SELECT ?elem {
    ?homedir nie:url 'file:///home/carlos' ;
             ^nfo:belongsToContainer ?elem
  }
  

  There’s harder ones like + and * that will require recursive selects, and there’s the negation (!) operator which is not possible to implement yet.

• We now have prepared statements! A TrackerSparqlStatement object was introduced, capable of holding a query with parameters which can be set/replaced prior to execution.

  conn = tracker_sparql_connection_get (NULL, NULL);
  stmt = tracker_sparql_connection_query_statement (conn,
                                                    "SELECT ?u { ?u fts:match ~term }",
                                                    NULL, NULL);
  
  tracker_sparql_statement_bind_string (stmt, "term", search_term);
  cursor = tracker_sparql_statement_execute (stmt, NULL, NULL);
  

  This is a long sought protection for injections. The object is cacheable and can service multiple cursors asynchronously, so it will also be an improvement for frequent queries.

• More concise SQL is generated at places, which brings slight improvements on SQLite query planning.

This also got the ideas churning towards future plans, the trend being a generic triple store as much sparql1.1 capable as possible. There’s also some ideas about better data isolation for Flatpak and sandboxes in general (seeing the currently supported approach didn’t catch on). Those will eventually happen in this or following cycles, but I’ll reserve that for other blog post.

An eye was kept on memory usage too (mostly unrealized ideas from the performance hackfest earlier this year), tracker-store has been made to automatically shutdown when unneeded (ideally most of the time, since it just takes care of updates and the unruly apps that use the bus connection), and tracker-miner-fs took over the functionality of tracker-miner-apps. That’s 2 processes less in your default session.

In general, we’re on the way to an exciting release, and there’s more to come!

Last evening I came back from the GNOME performance hackfest happening in Cambridge. There was plenty of activity, clear skies, and pub evenings. Here’s some incomplete and unordered items, just the ones I could do/remember/witness/talk/overhear:

• Xwayland 1.20 seems to be a big battery saver. Christian Kellner noticed that X11 Firefox playing Youtube could take his laptop to >20W consumption, traced to fairly intensive GPU activity. One of the first things we did was trying master, which dropped power draw to 8-9W. We presumed this was due to the implementation of the Present extension.
• I was looking into dropping the gnome-shell usage of AtspiEventListener for the OSK, It is really taxing on CPU usage (even if the events we want are a minuscule subset, gnome-shell will forever get all that D-Bus traffic, and a11y is massively verbose), plus it slowly but steadily leaks memory.

  For the other remaining path I started looking into at least being able to deinitialize it. The leak deserves investigation, but I thought my time could be better invested on other things than learning yet another codebase.

• Jonas Ådahl and Christian Hergert worked towards having Mutter dump detailed per-frame information, and Sysprof able to visualize it. This is quite exciting as all classic options just let us know where do we spend time overall, but doesn’t let us know whether we missed the frame mark, nor why precisely would that be. Update: I’ve been pointed out that Eric Anholt also worked on GPU perf events in mesa/vc4, so this info could also be visualized through sysprof
• Peter Robinson and Marco Trevisan run into some unexpected trouble when booting GNOME in an ARM board with no input devices whatsoever. I helped a bit with debugging and ideas, Marco did some patches to neatly handle this situation.
• Hans de Goede did some nice progress towards having the GDM session consume as little as possible while switched away from it.
• Some patch review went on, Jonas/Marco/me spent some time looking at a screen very close and discussing the mipmapping optimizations from Daniel Van Vugt.
• I worked towards fixing the reported artifact from my patches to aggressively cache paint volumes. These are basically one-off cases where individual ClutterActors break the invariants that would make caching possible.
• Christian Kellner picked up my idea of performing pointer picking purely on the CPU side when the stage purely consists of 2D actors, instead of using the usual GL approach of “repaint in distinctive colors, read pixel to perform hit detection” which is certainly necessary for 3D, but quite a big roundtrip for 2D.
• Alberto Ruiz and Richard Hughes talked about how to improve gnome-software memory usage in the background.
• Alberto and me briefly dabbled with the idea of having specific search provider API that were more tied to Tracker, in order to ease the many context switches triggered by overview search.
• On the train ride back, I unstashed and continued work on a WIP tracker-miners patch to have tracker-extract able to shutdown on inactivity. One less daemon to have usually running.

Overall, it was a nice and productive event. IMO having people with good knowledge both deep in the stack and wide in GNOME was determining, I hope we can repeat this feat again soon!

There’s been a rumor that I was working on improving gnome-shell on-screen keyboard, what’s been up here? Let me show you!

The design has been based on the mockups at https://wiki.gnome.org/Design/OS/ScreenKeyboard, here’s how it looks in English (mind you, hasn’t gone through theming wizards):

The keymaps get generated from CLDR (see here), which helped boost the number of supported scripts (c.f. caribou), some visual examples:

As you can see there’s still a few ugly ones, the layouts aren’t as uniform as one might expect, these issues will be resolved over time.

The additional supported scripts don’t mean much without a way to send those fancy chars/strings to the client. We traditionally were just able to send forged keyboard events, which means we were restricted to keycodes that had a representation in the current keymap. On X11 we are kind of stuck with that, but we can do better on Wayland, this work relies on a simplified version of the text input protocol that I’m doing the last proofreading before proposing as v3 (the branches currently use a private copy). Using an specific protocol allows for sending UTF8 strings independently of the keymap, very convenient too for text completion.

But there are keymaps where CLDR doesn’t dare going, prominent examples are Chinese or Japanese. For those, I’m looking into properly leveraging IBus so pinyin-like input methods work by feeding the results into the suggestions box:

The suggestion box even kind of works with the typing booster ibus IM. But you have to explicitly activate it, there is room for improvement here in the future.

And there is of course still bad stuff and todo items. Some languages like Korean neither have a layout, nor input methods that accept latin input, so they are badly handled (read: not at all). It would also be nice to support shape-based input.

Other missing things from the mockups are the special numeric and emoji keymaps, there’s some unpushed work towards supporting those, but I had to draw the line somewhere!

The work has been pushed in mutter, gtk+ and gnome-shell branches, which I hope will get timely polished and merged this cycle 🙂

Tracker is one of these pieces of software that get no special praise when things work, but you wake up to personal insults on bugzilla when they don’t, today is one of those days.

Several distros have been eager to push SQLite 3.20.0 still hot from the oven to their users, apparently ignoring the API and ABI incompatibilities that are described in the changelog. These do hit Tracker, and are only made visible at runtime.

Furthermore, there is further undocumented ABI breakage that makes FTS5 modules generated from pre/post 3.20.0 SQLite code backward and forward incompatible with the other versions. Tracker used to ship a copy of the FTS5 module, but this situation is not tenable anymore.

The solution then? Making it mandatory that SQLite >= 3.20.0 must have FTS5 builtin. The just released Tracker 1.12.3 and 1.99.3 will error hard if that is not the case.

I’ve just sent this mail to distributor-list:

Hi all,

Sqlite 3.20.0 broke API/ABI for Tracker purposes. The change described in point 3 at http://sqlite.org/releaselog/3_20_0.html is not only backwards incompatible, but also brings backwards and forwards incompatibilities with standalone FTS5 modules like Tracker ships [1], all of those are only visible at runtime [2].

FTS5 modules generated from SQLite < 3.20.0 won't work with >= 3.20.0, and the other way around. Since it's not tenable to ship multiple FTS5 module copies for pre/post 3.20.0, Tracker shall now make it a hard requirement that SQLite is compiled with builtin FTS5 (--enable-fts5) if SQLite >= 3.20.0 is found. The current Tracker FTS5 module is kept for older SQLite versions.

This change applies to Tracker >=1.12.3 and >=1.99.3. I don't know if any distro pushed SQLite 3.20.0 in combination with a Tracker that is older than that, but it will be just as broken, those would require additional patches from the tracker-1.12 branch besides the described change.

Please handle this promptly, wherever there's sqlite 3.20.0 without builtin FTS5 and/or tracker <= 1.12.2/1.99.2, there's unhappy Tracker users.

Cheers,
  Carlos

[1] Generated as described in
http://sqlite.org/fts5.html#building_a_loadable_extension
[2] https://bugzilla.gnome.org/show_bug.cgi?id=785883

So, if you see errors about “TrackerTokenizer” something, please contact your distro packagers. I’ll close further incoming bugs as NOTGNOME.

There’s a few plans in the boiler for Tracker:

Splitting core from miners

Tracker is usually deemed a “metadata indexer”, although that’s just half the truth. Even though Tracker could be essentially considered that in its very early days, it made a bold move back in 0.7.x to using Sparql as the interface to store and retrieve this metadata, where both the indexers and the applications using this metadata talk the same language.

So in reality, the storage and query language are useful by themselves. As per the plans, you’ll now have to distinguish between:
– Tracker the RDF store, and
– Tracker miners, the infrastructure and binaries to translate a number of entities (be it files or whatever) into Sparql, using the first acceptation for storage

Making standalone Sparql endpoints possible

At the time of moving to Sparql, Tracker was conceived as a global store of deeply interconnected data, Nepomuk seemed the most obvious choice to represent the data for indexing purposes and client isolation was basically left to 3rd parties. However times change, sandboxing is very present, and Tracker’s global store don’t help with it.

Based on initial work from Philip Van Hoof, quite some shuffling has been going on in wip/carlosg/domain-ontologies to make multiple Sparql endpoints (“database connections” in non-fancy speak) possible. This will allow applications to use private sparql endpoints, and possibly using other ontologies (“schemas” in non-fancy speak) than Nepomuk. The benefits are twofold, this will be a lot friendlier to sandboxing, and also increments the versatility of the Tracker core.

Switching to Semantic versioning

This change was suggested some time ago by Philip Van Hoof, and the idea has been growing in me. Tracker has usually had a longstanding backwards compatibility promises, not just in terms of API, but also in terms of data compatibility. However, bumping the version as per the GNOME schedule results in multiple stable versions being used out there, with the patch backporting and bug management overhead that it implies.

In reality, what we want to tell people (and usually do!) is “use latest stable Tracker”, often the stuff is already fixed there, and there’s no reason why you would want to stick to a stable series that will receive limited improvements. I do hope that semantic versioning conveys Tracker’s “later is better” stance, with some optimism I see it standing on 2.y.z for a long time, and maybe even 2.0.z for tracker core.

But versions are like age, just a number :). Tracker 2.x services and client side API will foreseeably be backwards compatible with 1.x from the reader standpoint. The Tracker core could be made parallel installable with 1.0, but I wouldn’t even bother with the high level infrastructure, 2.x will just be a better 1.x.

But this doesn’t mean we jump the GNOME unstable cycle ship. IMHO, it’s still worth following to let newly introduced code bake in, it just won’t result in gratuitous version bumps if API changes/additions are not in sight.

Code and API cleanup

In the more active past, Tracker had a lot code accretion while trying to get the buy-in, this included multiple miners, extensions and tools. But it was never the goal of Tracker to be the alpha and omega of indexing in itself, rather to have applications update and blend the data for their own consumption. Fast forward a few years and the results are mixed, Tracker got an amount of success, although apps almost exclusively rely on data produced by Tracker’s own miners, while most of these extensions are bitrotting since much of the activity and manpower went away.

Sam Thursfield started doing some nice cleanups of maemo/meego specific code (yes, we still had that) and making Tracker use Meson (which indirectly started tiptoeing some of that bitrot code). Several of these these extensions implementing Tracker support shall just go to the attic and should be done proper, it will at least be the case with nautilus thanks to Alexandru Pandelea’s GSOC work :).

But the version bump and code split is too good of an opportunity to miss it :). Some deprecated/old API will also go away, probably none of which you’ve ever used, there will be some porting documentation anyway.

Under public domain

There’s been lately lots of fuzz around Tracker as a security risk, as the de-facto maintainer of Tracker I feel obliged to comment. I’ll comment purely on Tracker bits, I will not comment on other topics that OTOH were not as debated but are similarly affected, like thumbnailing, previewing, autodownloading, or the state of maintenance of gstreamer-plugins-bad.

First of all, I’m glad to tell that Tracker now sandboxes its extractors, so its only point of exposure to exploits is now much more constrained, leaving very little room for malicious code to do anything harmful. This fix has been backported to 1.10 and 1.8, and new tarballs rolled, everyone rejoice.

Now, the original post raising the dust storm certainly achieved its dramatic effect, despite Tracker not doing anything insecure besides calling a closed well known set of 3rd party libraries (which after all are most often installed from the same trusted sources that Tracker comes from), it’s been on the “security” spotlight across several bugs/MLs/sites with different levels of accuracy, I’ll publicly comment on some of these assertions I’ve seen in the last days.

This is a design flaw in Tracker!

Tracker has always performed metadata extraction in a separate process for stability reasons, which means we already count on this process possibly crashing and burning away.

Tracker was indeed optimistic at the possible reasons why that might happen, but precisely thanks to Tracker design it’s been a breeze to isolate the involved parts. A ~200 lines change hardly counts as a redesign.

All of tracker daemons are inherently insecure!, or its funnier cousin Tracker leaks all collected information to the outside world!

This security concern has only raised because of using 3rd party parsers (well, in the case of the GStreamer vulnerability in question, decoders, why a parsing facility like GstDiscoverer triggers decoding is another question worth asking), and this parsing of content happens in exactly one place in your common setup: tracker-extract.

Let’s dissect a bit Tracker daemons’ functionality:

• tracker-store: It is the manager of your user Tracker database, it connects to the session bus and gets readwrite access to a database in ~/.cache. Also does notification of changes in the database through the user bus.
• tracker-miner-fs: It’s the process watching for changes in filesystem, and filling in the basic information that can be extracted from shared-mime-info sniffing (which usually involves matching some bytes inside the file, little conditionals involved), struct dirent and struct stat.
• tracker-extract: Guilty as charged! It receives the notification of changes, and is basically a loop that picks the next unprocessed file, runs it through 3rd party parsers, sends a series of insert clauses over dbus, and picks the next file. Wash, rinse, repeat.
• tracker-miner-applications: A very simplified version of tracker-miner-fs that just parses the keyfiles in various .desktop file locations.
• tracker-miner-rss: This might be another potential candidate, as it parses “arbitrary” content through libgrss. However, it must be configured by the user, it otherwise has no RSS feeds to read from. I’ll take the possibility of hijacking famous blogs and news sites to hack through tracker-miner-rss as remote enough to fix it after a breathe.

So, taking aside per-parser specifics, tracker consists of one database stored under 0600 permissions, information being added to it through requests in the dbus session, and being read by apps from a readonly handle created by libtracker-sparql, the read and write channels can be independently isolated.

If you are really terrified by your user information being stored inside your homedir, or can’t sleep thinking of your session bus as a dark alley, you certainly want to run all your applications in a sandbox, they won’t be able to poke on org.freedesktop.Tracker1.Store or sniff on ~/.cache that way.

But again, there is nothing that makes Tracker as a whole inherently insecure, at least not more than the average session bus service, or the average application storing data in your homedir. Everything that could be distrusted is down to specific parsers, and that is anything but inherent in Tracker.

Tracker-extract runs plugins and is thus unsafe!

No, tracker-extract has a modular design, but is not extensible itself. It reads a closed set of modules implemented by Tracker from a folder that should be in /usr/lib/tracker-1.0 if your setup is right. The API of these modules is private and subject to change. If anything manages to add or modify modules there, you’ve got way worse concerns.

Now, one of these extractor modules uses GStreamer, which to my knowledge is still the go-to library if you want anything multimedia on linux, and it happens to open an arbitrary list of plugins itself, that is beyond Tracker control or extent.

It should be written in rust!

What do we gain from that? As said, tracker-extract is in essence a very simple loop, all the scary stuff is handled by external libraries that will still be implemented in “unsafe languages”, rust is just as useful as gift paper to wrap this.

Extraction should be further isolated into another process!

There are good reasons not to do that. Having two separate processes running completely interlocked tasks (one process can’t do anything until the other is finished) is pretty much a worst case for scheduling, context switching, performance and battery life at once.

Furthermore, such tertiary service would need exactly the same whitelisted syscalls and exactly the same number of ways out of the process. So I think I won’t attract the “Tracker is heavy/slow” zealots for this time… There is a throwaway process, and it is tracker-extract.

The silver linings

Tracker is already more secure, now lets silence the remaining noise. Quite certainly one area of improvement is Flatpak integration, so sandboxed applications can launch isolated Tracker instances that run under the same sandboxed environment, and extracted data is only visible within the sandbox.

This is achievable with current Tracker design, however the “Tracker as a service” approach sounds excessive with this status quo, tracker needs to adapt to being usable as a local store, and it needs to go through being more of a generic SPARQL endpoint before.

But this is just adapting to the new times, Flatpak is relatively young and Tracker is slow moving, so they haven’t met yet. But there is a pretty clear roadmap, and we’ll get there.

So this is finally happening. The result of much work all through the stack from several dedicated individuals (You know who you are!) started lining up during the past few months and now is hitting master. Early in the cycle I blogged about stylus support being merged, based on the first version of the tablet protocols. Now I have the pleasure to declare GTK+ tablet support on Wayland feature complete.

As predicted in the earlier post, a second version of the protocol came through, bringing pad support for clients. What is a pad? It’s the set of buttons and other (often) tactile sensors that tablets have around the stylus-sensitive area. These devices are rather uncanny from an input management perspective: unlike mice/keyboard, those buttons/sensors don’t have an associated action that has been chiseled on the key or through decades of user interaction paradigms, they are rather meant to be user-mappable. Also, despite buttons being buttons and sensors being essentially one-dimensional axes, all resemblance with a mouse/stylus is purely coincidental, focus management is more similar to keyboards (actually, the same), and is not directly related to the stylus.

In GNOME, this action-mapping has been traditionally done in gnome-settings-daemon. X11 clients have been usually completely unaware of pad events, partly because of the oddities pointed out above. So g-s-d kept a passive grab on pad buttons and would translate those into keycombos. This has many shortcomings though, keycombos are far from standard, accelerators are translatable, … For Wayland, we have the opportunity to fix all these shortcomings, and make pads a first class citizen.

So the right thing to do here, if we want to univocally map actions to pad features, is delegating the action mapping to the client. The other side of the coin is providing proper feedback about the actions. In GNOME we have session-wide OSDs to display the pad mapping, and the wayland protocol has been tailored to cover an usecase like this, so applications can directly participate in filling in this info, this is how the end result looks:

* Note to supervillains: Actions are stubs

There’s also changes scheduled (I know I’m late, who doesn’t like pressure!) to gnome-control-center to centralize the management of styli for all known/plugged tablets, following the very nice mockups from Jimmac.

How is this exposed in GTK+?

gtk+ master introduced GtkPadController, it’s a GdkEventController subclass that will manage pad events, from one pad specifically, or for all at once. The controller takes a GActionGroup and a set of pad action entries, each defining an action name to activate, a simple example:

controller = gtk_pad_controller_new (GTK_WINDOW (app_window),
                                     G_ACTION_GROUP (app_window),
                                     pad_device /* May be NULL for all */);
gtk_pad_controller_set_action (controller,
                               GTK_PAD_ACTION_BUTTON,
                               1, /* Second button, they're 0-indexed */
                               -1, /* All modes */
                               _("Frobnicate file"),
                               "app.frobnicate-file");

And pressing that button on that pad will trigger the related action from the GActionGroup. The given label is what you end up seeing in the OSD. I expect this object to be eventually made useful on other platforms than Wayland (X11 is the first objective), although only the event-to-action mapping, and not entirely exempt of platform-specific issues.

This is going to be messy! why does every app need configuring? Why not a global action map?

The Wayland protocol aims to be intemporal, the trouble would only begin at defining a good enough initial set of global actions, and still not all actions might make sense for every app. This allows every application to implement pad actions, no matter how high or low level they are. Configurability is also optional, although the first apps that will surely come to your head will implement some, they already did for styli and other input features.

Is this for me?

Depends :), if you see your app might be involved in artists or designers’ workflow, it well could. Providing a sensible minimal set of actions to perform, even if hardcoded (think those you’d like shortcuts for, and were away from the keyboard), could help lots in integrating this previously neglected device, to the point that it might feel widely useful besides the niche drawing application case. By using GAction underneath, there’s also little left for you to test, you just need to ensure the action performs the intended effect when activated.

As for excuses go for blowing the lint off my blog, this is a pretty good one :). Wayland tablet support is something that got really close to being merged in 3.20, but the timing didn’t pan out in the end, wayland-protocols 1.3 included a tablet v1 protocol that went unused, till now. Now that we’re in 3.22, those bits are at last being merged. First came gtk+, which you can see working in these videos:

This improved tablet support brings some goodies (also on X11), there is a new GdkDeviceTool object that can used to track specific physical tools and provides the necessary info to do so in a persistent way (eg. across runs/reboots).

A note to all GTK+ app/widget developers

TL;DR: If you’re keeping track of motion events somewhere in your app, you want to check gdk_event_get_device(), see the bullet points below and check the best strategy for you.

One particularity in wayland tablet management is that whenever you have a tool in proximity to a tablet you’ll be driving a standalone onscreen pointer cursor, as opposed to the default behavior in X11 consisting of driving the same pointer than your mouse does. This difference is not moot, it doesn’t take too much thinking to realize that the user of a drawing tablet has a spare hand which could be using the mouse, so there’s in practical terms two pointers to react to. And there’s also setups where more than one drawing tablet is likely (the simplest one being a laptop with an integrated tablet plus a external one).

GTK+ is itself quite well prepared for these situations, as are all its widgets, it can’t account for widgets implementations elsewhere and signal callbacks in apps though.

But I’m not doing a drawing app, should I care?

Yes you should. Although tablets are mostly thought out for drawing, design, etc., ideally they should be able to drive the whole desktop without oddities, just as your mouse does. This can only drive to a pleasant experience with your app regardless of the input devices of choice.

If there’s somewhere in your app where you’re tracking pointer events over time (drawing is obviously an example, but so is rubberband selection, any kind of drag-to-move, etc…), there’s a chance that it could get confused by events with different gdk_event_get_device()s coming to the widget in question. There’s several things you might do:

• First and foremost, look into GtkGesture and its several subclasses, it’s handled transparently there, and there’s a good chance that one of those (or a group) will fit your case
• Maybe it’s sufficient a “first to come gets served” basis for the specific purposes of your operation? You can use gtk_device_grab_add/remove with that device, or check gdk_event_get_device() yourself on all events.
• Think it’d be cool to handle the simultaneous pointers and want to go the extra mile? Again, track gdk_event_get_device(), and keep per-device state.

…And there’s more to come

There’s big plans for mutter in this cycle so tablet support will not be immediately merged, but be assured it will happen in due time. Gears are also grinding in order to have a version 2 of the tablet protocol, it will mostly focus on bridging pads (the sets of buttons and sliders along the sides of some tablets) to wayland clients, something which I expect to be an outstanding improvement compared to X11, so… stay tuned!