Carlos Garnacho

With 3.14 almost out of the door, it seems like a good opportunity to blow the cobwebs of this blog and highlight some shiny new features I was involved in during this cycle:

Gesture support in GTK+

It’s finally happening :), GTK+ 3.14 brings in infrastructure to handle gestures. Maybe the word “gesture” is automatically associated to “multitouch”, but actually this infrastructure is meant to deal with all kinds of pointer/touch input, and as such is used fairly intensively now within GTK+ itself, so even mouse users will be unknowingly using this.

These gesture objects are of course readily available for applications too. Individually, these are quite simple in essence, but can be easily stitched together to compound higher-level behavior. So far, eog and evince (and by extension gnome-documents) have bitten the bullet and now handle some of the gestures you’d expect on touchscreens there, the API documentation and HowDoI are available for anyone wanting to follow.

Gesture support in gnome-shell

Just to feed the cabal claiming that gnome-shell is designed for tablets, I also managed this cycle to add gesture infrastructure in mutter, so it is able to pre-process touch events before applications do, these events are then handled through ClutterGestureActions, or “rejected” and eventually handled by the application. This has been put to use in gnome-shell, making some actions readily available through multitouch.

Edge swipes

Showing the overview

Switching workspaces

Switching apps in a workspace

Note: Recorded with the help of this, bug #732367 is yet another 3.16 todo item…

Freebase support in libgdata

This one feature has certainly went underpublicited, and I found myself with little time to make use of it :(, but I nonetheless find that very interesting things can be done with this. Freebase is a community-maintained knowledge base (currently backed by Google), as seen on its homepage it is extremely wide in topics (some better covered than others), and has a very well defined ontology, think of it as a companion to Tracker on the web.

There are dedicated methods for the most usual ways to query data (search, lookup on topic…), but additionally Freebase offers a powerful MQL query method. MQL is very analogous to SPARQL, with the main difference that it’s based on JSON. All together allows for querying in very creative ways from very various data, a few examples being:

• The mandatory “query for movie/album info” example, actually these topics are the best covered.
• Fetching stock images for movies/cities/landmarks/directors/…, you name it.
• Looking up monuments close to a geolocation.
• Getting direct links to Wikipedia, in your language.

Looking forward for 3.15

It is almost time to celebrate, but I evidently won’t sit twiddling my thumbs :), a few items I’d like to tackle on the next cycle are:

• During the 3.14 cycle I got briefly started on adding optional gesture integration to GtkStack and a new “tabs” widget, now it sounds like a good time to resume. I would also like to make gestures used integrally for event handling in GTK+ (we’re already just a few widgets away from that goal)
• There’s a few gaps still left to solve on handling touchpad gestures, which I’d like to get closed ASAP, at least for touchpads handling >2 fingers on X11.
• Improving gnome on Wayland. I merely got started this cycle adding DnD/clipboards support to GTK+ and bringing touchscreen behavior on mutter more or less on par to X11’s. There’s a few input details that need shuffling so they’re done in the same place on X11/wayland (pointer cursor visibility, device mapping…), and I hope the timing to be right to bring in a sort of tablet support (libinput and wayland protocol details have been shaping up despite my times on, most times off help, thanks Peter, Lyude, Jason et al!), I will be putting my hacking efforts wherever it’s necessary to make this happen.
• WebKitGTK+ could be definitely made friendlier on touchscreens, additionally to the DOM touch event support it already does, it would be great to handle touch scroll/pinch/zoom as you can see in other pure GTK+ apps now.

Last week the developer experience hackfest took place in Berlin. Quite some blogging has happened during and since, nonetheless this was a pretty good excuse to blow the cobwebs off my blog and chime in too :).

The event was quite intense, plenty of discussion happened around the GNOME platform, developer tools, API documentation generation, and introspection. My main point of interest there was GTK+, it was great to see a roadmap shape up, and a consensus about the gestures branch (more about that below). I also offered to look into doing something that could replace GtkNotebook in the future, which I started hacking on during the hackfest. I also had a nice opportunity to chat with Philip about Freebase support in libgdata, and solve a few questions around Tracker.

Other than that, nice chatting, and a very fine choice of places for food and beverages. Many thanks to Chris Kühl and the Endocode guys for making this as much enjoyable!

Gestures

One point in the GTK+ meeting agenda was gesture support. The gestures branch (which has had a somewhat intermitent history during the last ~2.5 years) has lately shaped up into something that was agreed mergeable for 3.14. This branch introduces “gesture” objects that abstract away all the intricacies of (multi)touch management. These objects handle events, tracking individual touches, when gestures enters in a “recognized” state, high-level signals will be emitted. This has been made to work parallelly to regular event delivery, in order to allow for different degrees of interoperation with handlers there, and making single-touch gestures to work with pointer events is just a call away.

There is of course a need of cooperation between gestures, both within a widget, and across the widget “stack” that’s receiving events for each individual touch, each of those will be triggering a number of gesture objects as the touchpoint moves. To cater for that, gestures have a none/claimed/denied state on each tracked touch, and changes in those are communicated across the stack (possibly causing cancellation on child widgets also handling the touchpoint, or cancelling other same-level gestures that previously claimed the sequence).

Intra-widget, some simultaneity may be wanted when handling touchpoints (eg. long press and nth-click handling), gestures may be grouped so those share the same state and can act upon the same events, a widget can have multiple mutually-exclusive gesture groups (eg. scrolling and “switch page” panning on a scrolledwindow).

So that is it! in a very tiny nutshell, the API added in the branch is documented, that will be a more accurate source :). Now, a video showcasing this, unedited because Pitivi didn’t get along today. A 42 seconds journey from well-known lands to barren plains of insanity, made manageable thanks to this branch:

The branches for eog and evince are available too, for anyone willing to play soon with this. But as said, hopefully everything will be soon in master!

The facts narrated here happened on early November, but I’m apparently blog-constipated, hence the extra month…

So, after an overall of 6.5 years working on Lanedo (and former Imendio), I made my last day there on October 31st. After all that time, I consider it a privilege having worked with them, they’re a smart bunch and I’m very sure they’ll be doing fine

After that? Shiny new times, this happened:

On November 1st I joined Red Hat! And so far it’s been an incredible experience. With them, I’ve spent much of my time tinkering, surrounded by tablets, touchscreens and other input devices, and will be doing so in the foreseeable future, so expect bursts of activity around the input area :).

Around early 2012, I made some rough experiments around input and event delivery that required nothing but Glib, X and cairo. It was only a bit later, when I had a basic event delivery and rendering model, when I started thinking on how would have windowing subsystems in toolkits evolved if we had all the tools and lessons learnt that we now have.

Fast forward some months of times on, times off development, and I think this turned into something worth sharing, all say hello to Mechane. On that time, ideas flowed, many were ditched, wayland was fully embraced, and this experiment is eventually turning into a lean codebase:

https://github.com/garnacho/mechane

What is Mechane?

Mechane is a concise wrapper to the windowing system, with all expected modern features integrated in the core. There’s several widgets, ranging from simple to complex, exercising these features. Even though it internally has a backend abstraction, there is only a wayland implementation.

Some gory details

The basic building block in Mechane is a MechArea, as the codebase was in principle built to check input-related concepts, it was made so elements could be reused most readily, so a micro-widget approach was taken, similar to Rapicorn[1].

Under this approach, simple MechAreas do as little as possible, and those work as building blocks to create growingly complex composited MechAreas. There is an emphasys on using GInterfaces for the most usual data exchange with an UI, as composite areas have facilities to delegate full interfaces on child areas. This all allows for making composited areas that are fairly opaque on the outside, while the little inner pieces focus on their business.

Transformations are also dealt with as a core feature, any MechArea can be assigned a transformation matrix. The coordinate space on events is fully virtualized (starting on 0,0 on the area receiving the event), there’s though ways to translate between coordinate spaces whenever needed. One can think transformations is mainly necessary for effects, scrolling is the most notable example of transformations in Mechane though.

On this design, the basic MechArea that does text display/edit will be terribly ubiquitous if used as the foundation for entries, labels, text views and whatnot. As such quite some emphasis was made at having something that scales very well, and the result has gotten really nice: MechTextView can handle massive ammounts of text, letting you instantaneously scroll anywhere in the buffer [2], no CPU grinding happens on the background on any situation.

And of course, wayland offers some possibilities that weren’t as readily available in X11, mechane so far handles SHM and EGL surfaces, even though this is all quite transparent thanks to cairo (unless of course you want to mess around with that)

But this all started around input! there’s a collection of gesture controllers and incipient experiments around making those cooperate across a hierarchy, there’s though interesting puzzles around grab transfers and cancellation that are still subject to investigation.

This all makes a very nice result, every piece is very self-contained, in a way that many things can be reused or made to work elsewhere, eg. a “complex” widget like a scrollable view came out like a piece of cake. As for rendering performance, going through software rendering it beats comfortably the 60fps mark in most situations on now oldish laptops.

In such early stages, there’s of course quite some open fronts: first and foremost is input handling, also some wayland features are still unsupported, like subsurfaces and clipboards, and the techniques used for validation-less textviews is worth investigating on icon/list views.

Of course

Kudos to Kristian Høgsberg and all wayland/weston/cairo folks. I’m happy how lean the Mechane is turning out to be, but it could only be so lean thanks to their work.

Going to GUADEC!

For the 11th consecutive time, I’m attending GUADEC! Thanks to Lanedo I’ll make it to Brno in 1.5 days. Aleksander and I will be talking about Tracker. So if you want to talk about Tracker, Mechane, GTK+ or input handling, I’ll be around, see you there!

[1] Yes, this is the second toolkit-thingy from your fellow lanedians, there must be some irony in it.
[2] Well, somewhere between 500 and 700MB of text it goes from “snappy” to “responsive” here due to other arising bottlenecks, still more text I can read without my eyelids falling off.

After quite some time without touching Tracker code, last week I finally could get back to a branch that’s been sitting there for some time now. On fts4, sqlite requirements have been updated to >=3.7.9 so we can stop compiling our custom FTS module and start using what comes with libsqlite.

What does this mean? Internally there’s less code on our plate (and non-stale), and external content support in FTS tables brings us no performance nor file size decreases. plus we get all recent hot improvements in sqlite releases for free.

A bit more on the user point of view, a feature that became possible with this swich is the support for fts:snippet(), which you can use in SparQL queries to get snippets of the matched text:


$ tracker-sparql -q "select nie:url(?file) fts:snippet(?file) fts:rank(?file) where { ?file a nfo:Document ; fts:match 'reference' } order by desc (fts:rank(?file)) limit 3"
Results:
file:///home/carlos/Documents/Papers/pdf_reference_1-7.pdf, ...Reference Streams G8.1872911 Cross-Reference..., 46.0
file:///home/carlos/Downloads/addison.pdf, ...GLU are described in the OpenGL
Reference Manual. The more useful GLU..., 40.0
file:///home/carlos/Downloads/ThesisHo.pdf, ...A8 ]+ ) is also included for
reference. In the third experiment, we apply..., 40.0


So its easier to the eye, tracker-needle search tool now also shows snippets where available, providing a nice context for the matched content

Remember that FTS searches apply to any property that’s specified by the ontology as tracker:fulltextIndexed, you can run this to find out:

tracker-sparql -q "select ?prop rdfs:label(?prop) tracker:weight(?prop) where { ?prop tracker:fulltextIndexed true }"


There’s also the possibility you had no idea what I’m talking about :), If desktop semantic search still tickles your curiosity I’ll point you to the fine gathered documentation about Tracker.

This work was kindly sponsored by Lanedo.

So, after a few strives during the last year, the multitouch Xorg patches were posted and merged to master last month, making multitouch available in the upcoming Xorg release. This turns the multitouch GTK+ branch into a suitable candidate for GTK+ 3.4, which obviously deserves a video demoing what’s up there:

Hopefully soon in master, very soon…

Earlier today I’ve pushed gedit-cossa, a plugin for gedit to help writing CSS for GTK+ themes, it is able to display a number of samples, loaded from GtkBuilder files.

Here’s a video demonstrating how it works:

Cossa is still in pretty early development stages, immediate plans include:

• Hooking CSS parsing errors to the gedit view
• Adding a lot more samples, these should range from simple examples (basic widgets in different states) to complex (basic main window sample, preferences dialogs, …)

Anyone is welcome to help, specially in the second point, as it is fairly straightforward to add new samples.

Along the last days/weeks, I’ve gave myself the oportunity to tinker with XInput 2.1 in GTK+, now that the core concept of the involved changes seem settled (although corner cases are still being discussed).

Thanks to the previous port to XInput 2.0, handling multitouch events has been fairly easy, by enabling GDK_TOUCH_MASK events on your widget you can receive GDK_TOUCH_PRESS/MOTION/RELEASE events containing a touch ID, which is unique at that time for a touch event stream, so enabling that you can effectively receive events from simultaneous touches.

In addition, one can create GdkTouchClusters (related to a GdkWindow) and add touch IDs to it. from that point on, any update on these touch IDs stops sending individual GDK_TOUCH_MOTION events, instead sending GdkEventMultiTouch events, containing all latest events for the touch IDs in a GdkTouchCluster.

I’m pretty happy with the resulting API, althought there are some internal details left to improve, my main gripe currently is that implicit grabs in CSW still happen per device, this means no multi-widget multitouch yet within an app (also posing interesting problems with keyboard events redirection and explicit grabs). Another big item is looking into integrating this with gestures, as direct manipulation and gesturing need to go hand in hand, so that branch could be discontinued in favor of this one.

As I’m pretty lazy, and the results would be strikingly similar to what I previously posted, I won’t post any video, so you’ll have to settle for a screenshot:

This is sitting now in the xi21 branch. If you just compile this you won’t see much working though, the demo only reacts to touch events, which are only sent if XInput 2.1 is there, which requires compiling certain Xorg modules branches, and having a multitouch device at hand.

Fixing the default GNOME3 theme

Now that most of GTK+ widgets are using GtkStyleContext, I finally got onto improving the default gnome3 theme/engine, I finally chose to rip off the tiny clearlooks engine bits I needed to demonstrate how minimal engines can complement and extend the current CSS theming features. So things are getting closer to the mockups:

There are some things that still need to be improved, column headers and expanders most namely, but things should be mostly there pretty quickly. Even though, hands and eyes are most welcome on both the CSS file and the engine.

A gestures interpreter for GTK+
Despite what some might think, my day to day work carries me quite far from theming land, during the last week I got the oportunity to start development on a gestures interpreter for GTK+, a pretty interesting (read: wacky ) piece of dynamic programming, that seems to work pretty well for its builtin gestures (Video here, at the moment it handles swiping in the 4 directions, plus circular swipe in both directions)

This code lives at the moment in a GTK+ branch, including a testcase. The gestures interpreter at the moment handles individual pointer movements separatedly, although it may be improved over time to handle multiple input pointers, as the interpreter itself is private and widgets just get an enum for the gesture type.

As the stock gestures could be stored in terms of coordinates or vectors, future plans include having these stored in a mmap()-able format, plus having an editor to create new gestures.

For those building gnome3 frequently from git or paying attention to gtk-devel, this might be old news, but still worth mentioning… The gtk-style-context branch has landed in GTK+ master, which will quickly unveil a powerful theming system, powered by CSS files.

Although some apps (mostly these implementing widgets themselves) might undergo some work to adapt to this, there are API docs and migration documents (mostly courtesy of Matthias, who’s done an excellent work) that should ease the task.

Besides the GTK+ work, there is also both a gnome3 theme and engine using the new stuff. Things are still ongoing here, as many things can be leveraged into CSS, which would leave the engine very little things to do. Andreas started looking at some point into the branch, so he might have some things ready in that field :).

But not only that! Carlos Garcia Campos (KaL) has been doing good progress on webkit-gtk so no backing widgets creation is needed, and Paolo Borelli has ported GtkSourceView as well, awesome!