GNOME and gestures, Part 2: HdyLeaflet

This is part 2 of a mini-series. Part 1, Part 3.

Shortly after the WebKit gesture was merged, I started experimenting with making this gesture more widely available. The first step was porting it to Vala and decoupling from WebKit. Since I wrote this part of the gesture tracker from scratch anyway, it was simple and straightforward. The resulting playground project was also used as a convenient place to quickly iterate on the WebKit gesture itself. Later I also reimplemented rendering to match the WebKit one. Here’s how it looked at various points of time:

Other than that, I used the swipe tracker to make a few more demos for Tobias Bernard:

Check out his GUADEC talk showcasing the second demo! :)

At the same time, I started integrating it into libhandy by supporting back/forward swipe in HdyLeaflet. And there I hit four problems:

1. Transitions and `visible-child`

A folded HdyLeaflet, just like GtkStack, shows one of its children at any given moment, even during child transitions. The second visible child during transitions is just a screenshot. But which child is “real” and which is a screenshot? Turns out the real child is the destination one, meaning the widget switches its visible child when the animation starts. It isn’t a problem if the animation is quick and time-based, but becomes very noticeable with a gesture. Additionally, it means that starting and cancelling a gesture switches the visible child two time.

One solution would be only switching the visible child at the end of the animation (or not at all if it was canceled). The problem is that it’s a major behavior change: applications that listen to visible-child to know when to update the widgets, or sync the property between two leaflets will break.

Another solution would be to draw both children during transitions, but it still means that visible-child changes two times if the gesture was canceled. The problem here is similar: applications wouldn’t expect the other child to still be drawn, but at least it’s just a visual breakage. And it still means that starting and canceling the gesture would mean two visible-child changes.

The second solution may sound better, and yet the current WIP code uses the first one.

2. Visuals

Leaflet had many issues in this area, such as over transition not making sense spatially and bottom widget being visible through the top widget. Additionally, Adrien liked the drop shadow and dimming in WebKit and the demo and wanted to have it in leaflet as well. :)

The first issue was solved by splitting the transition into over and under and clipping the bottom child. Similarly, I implemented shadow and dimming, though it’s pending on those transition types for mode transitions being merged first, so that the shadow can also be added to those, so that it’s consistent.

I’m also not happy with how the dimming and shadow are implemented, neither here nor in WebKit: it’s custom drawing with hardcoded values. Ideally, this needs to be controlled from CSS somehow. GTK itself uses gadgets for things like this (for example, the overshoot effect in GtkScrolledWindow), but that API is private. Having dimming and drop shadow widgets is an overkill, at least until GTK 4 arrives and makes GtkWidget instantiable. Maybe foreign drawing could work…

3. Syncing animation

Often, GTK applications have two leaflets: one in the window’s content area and one in titlebar. Their visible child is always changed at the same time, so it looks like they are one tall leaflet spanning both titlebar and content. This still needs to work with the gesture. And while it’s easy to make nice-looking throwaway demos that do this, syncing actual HdyLeaflets has to be a proper API.

Initially I suggested what I thought was a nice solution with having swipe tracker as a public object and connecting multiple widgets to it. Benjamin Otte and other people immediately pointed out many problems with it, so I researched how other platforms do it. The answer is simple: most platforms don’t. :)

Android has a rather silly way to sync multiple widgets together, but it’s rarely needed, as app bars are just widgets, so they can be packed into a ViewPager without a need to sync two pagers together.

Another constraint is that the solution must not expose animation progress as a write-able property, so it must not be possible to set this value to something arbitrary and get the transition stuck.

4. Interaction with `GtkScrolledWindow`

GTK event propagation works in two phases: capture and bubble. Widgets can connect to event signal and receive events on bubble phase. Then they return a value to either stop the event or propagate it further. More recently, GTK added various event controllers that allow choosing the phase where they run. With GTK_PHASE_CAPTURE it’s possible to handle events on the capture phase… But their signals don’t support fine-grained stopping/propagation, i.e. don’t have return values (they do in GTK4 though).
All in all, it means that there’s no way to get an event on capture phase and stop it atbitrarily…

Except there is, it’s private and it’s used by GtkScrolledWindow. This widget captures scroll events and stops some of them. For example, if the scrolled window has a vertical scrollbar, but not horizontal, it stops vertical scrolling events and propagates horizontal scrolling. This is harmless, but it also always stops events with is_stop set to TRUE, meaning a leaflet containing a GtkScrolledWindow will get stuck at the end of the gesture. So every single way of receiving events fails in a different and exciting way.

This last issue made me hate life and put the project on a long hiatus.

More demos

A while later while doing another demo (more on that in the next post) I discovered the horrible workaround: the private function for capturing events in GTK has a very simple implementation, so it’s easy to set this handler manually. And of course, with this workaround it just works. This solves the issue #4.

For the issue #3 I made a crude solution similar to already existing HdyHeaderGroup: HdySwipeable and HdySwipeGroup. It’s an RFC at this point, so criticism is welcome.

That allowed me to make a fully working (though still buggy) prototype of swipeable leaflet:

(Yes, that’s a bug there on 0:21)

Another visible problem in the video is that HdyHeaderGroup showing and hiding buttons doesn’t really work with the gesture. One possible solution here would be to show all buttons on all headerbars when folded, but that would once again involve an API break.

The (still very messy) code is here. Even though it’s not ready yet, Shortwave app already makes use of it:

This also uncovered a crash when a leaflet is created in unfolded state. Oops.

Thanks Felix Haecker for testing, and once again Tobias Bernard for feedback and suggestions while iterating on it.

GNOME and gestures, Part 1: WebKitGTK

This is part 1 of a mini-series. Part 2, Part 3.

Re-publishing, the original post is too old to show up on Planet GNOME at this point. It was published on August, 8th initially.

I’m a big fan of responsive touchpad gestures. For the last half a year (mostly January, February and during the summer) I’ve been working on improving gestures in many areas throughout GNOME. In this series I will do a (belated) overview.

Back/Forward Swipe

Late in the 3.32.x cycle, I saw a commit by Jan-Michael Brummer adding a back/forward swipe to Epiphany. It was really nice to finally have gestures, but it didn’t have any visual feedback. Less importantly, the direction was reversed, as if when scrolling with Natural Scrolling being off. I wanted to give a shot at improving it.

A proper gesture would have to “stick to finger”, showing screenshot of the previous or next page during the gesture, more or less what Safari does on macOS. Specifically, Epiphany would have to take screenshot of every page that is added into back/forward history, show it while the gesture is performed, then continue showing it until the next page loads enough to replace it. Unfortunately, this isn’t really possible to achieve in Epiphany itself: while WebKit does provide API to take snapshots, there’s no way to know when the previous/next page has loaded “enough”.

So I started looking into WebKit instead, where I found out that Safari’s gesture is actually implemented right there! Most parts are present, but the code was not cross-platform. So I started slowly adapting it for GTK. For the most part, the reusable code was a large part of the back end: page snapshot store and snapshot removal logic. That code is now shared between the platforms. The other parts, like actual event processing and drawing, had to be written from scratch.

One interesting detail about the gesture is that it doesn’t actually use gesture events! Libinput defines swipe gestures as synchronous movement of three or more fingers in the same direction. Mac gesture API is even more strict: it’s three fingers only, with four-finger swipes being reserved for the OS. But the gesture uses two fingers, how is this possible? Turns out it actually uses scroll events instead. (That’s also why it works with Magic Mouse in macOS, even though the code does not special-case anything for it)

When using scroll events, one has to be very careful. Touchpads generate scroll events continuously, in GTK it means that these gestures have GDK_SCROLL_SMOOTH scroll direction. At the very end of the scrolling, there will be a special event with is_stop field set to TRUE, which is used as a signal to start kinetic scrolling or, in our case, to end a swipe.

But there are other input devices, for example, mice. Most mice have “clicky” wheels that generate scroll events with direction instead of deltas. These events are impossible to use for swipes, so there’s no point in even trying to handle them. But there are also mice with freely scrolling wheels which generate the same events as touchpad, except there’s no event with is_stop == TRUE at the end. This means that they can be used to start a swipe, but it will get stuck as soon as the wheel stops. So, these mice have to be skipped too. Then there are touch mice where scrolling probably works same as on touchpad. I suspect swiping can work very well with them, same as it does with Magic Mouse on macOS, but there’s no way to distinguish these kinds of mice, at least as far as I know, so I had to disable it for any mice.

Another problem is that in order to not interfere with actual scrolling, the gesture controller must check whether it is possible to scroll the page first. Maybe there’s still space to scroll, maybe the page intercepts the scroll events. Then there has to be a threshold so that it’s hard to accidentally trigger the gesture. Thankfully, this part is shared with the Mac gesture. :)

A side effect of using scroll events is that this gesture still works with X11 and/or older semi-mt touchpads for which Libinput normally does not support any gestures.

So, now the gesture sticks to fingers. But there’s still an important bit missing: a snap-back animation. In order for a gesture to feel natural, it should snap back smoothly as soon as you lift your fingers, respecting the momentum. This was a lot easier to do than I expected, and after a few iterations applying Tobias Bernard’s suggestions I had an animation that I’m very satisfied with. How it works:

The animation uses easeOutCubic interpolation
Duration is naturally calculated as remaining distance divided by velocity, or if the velocity is 0, by a constant value instead
After that, duration is multiplied by 3, matching easeOutCubic derivative at t=0. This ensures that initial velocity is same as it was before lifting the fingers
Finally, the duration is clamped into [100ms, 400ms] range. This ensures that it’s never too slow or too fast, while still allowing it to respect momentum when possible
If the page was swiped less than halfway through the window, there’s a small velocity threshold. If fingers are lifted when not moving, the gesture will be canceled and the page will smoothly slide back. On the other hand, if the page was swiped more than half way through, just lifting the fingers would finish the gesture, and one has to specifically flick back to cancel it
If one starts swiping again while the animation is going, it will actually be stopped. This allows to continuously grab and release the page

Interestingly, Mac has a helper function that takes care of all this, and WebKit makes use of it.

Finally, the gesture should look nice. On Mac it uses CoreAnimation for drawing; WebKitGTK has to use Cairo. Since I didn’t have any mockups to work with, I reused Apple’s visuals, consisting of a dimming layer and a long subtle gradient for drop shadow, intending to replace them with something else later. But everybody whom I showed it liked it, so I left it as is.

The end result is that since version 2.24.0, WebKitGTK optionally supports 2-finger swipe gestures on touchpad.

Unfortunately, I was a little too late to enable it in Epiphany 3.32.0, but it was merged into 3.32.1 nevertheless. In 3.34.x, Yelp and Devhelp will also support it. Additionally, it’s enabled in Eolie and Odysseus browsers.

A bit later I also added touchscreen support. That was easy, because WebKit literally generates scroll events for touch scrolling, so it was simply a matter of feeding those events into the gesture controller. Additionally, I had to implement canceling, as all touchscreen gestures have to support it. Since scrolling on touchscreen is 1-finger swipe, the gesture is performed the same way.

This will be available in upcoming WebKitGTK 2.26.x, corresponding to GNOME 3.34.

This can be very disruptive feature in many cases, such as authentication widgets, so applications wanting to use it have to opt in by changing the value of this property.

Pinch Zoom

A smaller change was getting pinch zoom gesture to work on touchpads. Since this gesture was already available on touchscreens, it involved simply feeding touchpad gesture events into the gesture tracker, but the performance is severely lacking on heavy pages. Speeding it up is unfortunately still above my skill level. :)

Pinch zoom is enabled unconditionally, so it already works everywhere where WebKitGTK is used, including but not limited to Geary and documentation view in GNOME Builder.

I want to say thanks to the following people:

Michael Catanzaro and Carlos Garcia Campos for code review and helping me with understanding WebKit codebase
Tobias Bernard for testing and numerous design suggestions
Jonas Dreßler for testing and feedback, especially on a touchscreen

Part 2 here.

GNOME Games 3.34

A year ago, Adrien Plazas stepped down as a maintainer, so Games 3.32.0 was released without an accompanying blog post, since I didn’t have a blog at the time. Now it’s time to make up for it with a blog post about 3.34.0.

GSoC and savestates

As part of his GSoC project, Andrei Lişiţă implemented a savestate manager.

Savestates are a common feature in game emulators, that work similarly to snapshots in virtualization: emulator takes a full snapshot of RAM and storage, which can be loaded later to restore the game to the same exact state it was in when saved.

The app has supported savestates for a long time: when you exit a game, a savestate is created. Then when you run it again, Games offers to restore that savestate or reset the game. However, there was no way to manage savestates during the game, or to have more than one savestate at a time.

Now we have a shiny new sidebar for managing savestates and can save and load them on demand. This can be used for saving memorable moments in games, or for cheating your way through difficult games via saving and reloading every time you make a mistake.

There is still room for improvement, for example, there is no way to use savestates with gamepad right now.

Since directory layout is different now, existing data is automatically migrated on the first run. This is a one-way process, downgrading to 3.32.1 after running 3.34.0 is not possible!

Nintendo DS screen layouts

GNOME Games supports running Nintendo DS games since 3.30. However, the system’s two screens make it awkward to play without having a screen in portrait orientation. While ideally we want retro-gtk to support support rearranging screens, for now I implemented this using options of DeSmuME core:

So far we support a vertical mode, two horizontal modes and a single screen mode, corresponding to DeSmuME’s top/bottom, left/right, right/left and quick switch modes. While the first 3 modes work exactly the same as in RetroArch, single screen mode has some improvements: we remember and restore the currently viewed screen, provide a visible button for switching between the screens and have separate keyboard shortcuts for that (it’s using top only and bottom only internally, not quick switch).

Naturally, a downside is that it only works with DeSmuME and DeSmuME 2015 cores.

At first it was a global setting. This led to some problems, such as squished screenshots when loading the game if the current layout doesn’t match the one the game was played with the last time, which was made even more apparent after Andrei’s savestate work landed, showing the squished screenshots even more prominently.

Because of that, I reworked the feature to store the screen layout inside savestate metadata instead of a global setting. While this required a fix in retro-gtk, screen layout now can be different not just for every game, but even for every savestate, meaning previews always perfectly match the loaded game.

Platforms instead of Extensions

For a long time, Games had an “Extensions” page in preferences, showing all the installed plugins. This was one of the more confusing parts of the app:

There are many problems with this page: many people assumed that the listed platforms were the only platforms we support, even though there are 24 more platforms that are supported directly, not via a plugin. Additionally, “platform support”, via plugin or not, means that Games can list games from this platforms in collection view. It does not automatically mean that the app can run those games, that would also require the corresponding libretro core to be installed. And even then, some games need firmware to run. None of this is really obvious from the page.

All in all, this means that Games can run NES and Game Boy games, even though they aren’t listed on the page, and at the same time it cannot run DOS or Sega Saturn games, even though they are listed, which makes no sense unless you’re involved in development of the app.

Even more confusingly, the modules that the page lists are actually called “plugins” internally, not “extensions”.

To solve these problems, this page has been removed and replaced with a Platforms page that lists supported platforms, lists the currently used core for libretro platforms and grays out the platforms without cores, so that it’s visible which games can and cannot be run.

While our official build on Flathub ships only one core per platform, it’s possible to have multiple cores for a single platform. In that case, the page will allow to choose the core to run the games with. I hope the new page will be a lot more useful.

Backup and Restore

Another feature that landed during this cycle is backing up and restoring for savestates, courtesy of Adwait Rawat.

These changes include a page in preferences that allows to export all the savestates as one archive, or to restore an existing backup. This can be useful, for example, to move data to a different device.

Goodbye, developers view

Developers view was implemented along with Platforms view by Saurabh Sabharwal as part of his GSoC 2018 project. While Platforms view was very successful, Developers view used thegamesdb metadata, which wasn’t very reliable. Often, it mixed developers and publishers, or duplicated the same developer multiple times with slight variations. And then, thegamesdb changed its API, so for the last few months, this view has been completely empty. Because of that, I went ahead and removed this view:

Adaptive UI

I’ve been slowly making the UI adaptive for the last year. 3.34.0 finally marks the point where the app can run on a phone:

Games 3.34 on mobile, platforms — Platforms

Games 3.34 on mobile, viewing a platform — Platform details

Games 3.34 on mobile, playing a game — In game

Games 3.34 on mobile, savestates — Savestates

Games 3.34 on mobile, preferences — Preferences

Games 3.34 on a phone, Backup & Restore page — Backup & Restore

However, there is still a very important bit missing: touch controls to actually play games without a gamepad or keyboard.

Better Fullscreen

In 3.32.1, headerbar in fullscreen is hidden, but shows up on any cursor movement. This can be annoying when playing a mouse-heavy game (for example, on Nintendo DS), so in 3.34.0 the headerbar can only be revealed by pushing the top of the screen, similar to the behavior in Epiphany and other apps.

Miscellaneous Changes

Andrei added an error message that shows up after opening a non-game file:

Media switcher has a dropdown arrow now:

Cursor now autohides in windowed mode after 3 seconds of inactivity, just as in fullscreen.

Game covers aren’t blurry on HiDPI screens and aren’t darkened anymore.

Thanks to all the contributors who made this release possible!

Getting Games

As always, the latest version of the app is available on Flathub.

Games and GSoC 2019

GNOME Games has been participating in Google Summer of Code for many years, and this one is no exception. This time Andrei Lişiţă a.k.a. Yetizone was implementing a savestate manager.

Andrei’s work involved redoing $XDG_DATA_HOME/gnome-games/ directory layout, writing a migrator for existing data, reworking the app to support having multiple savestates at once, implementing on-demand loading and saving, and implementing the UI.

Check out his blog posts for more information about the project:

See also his lightning talk at GUADEC 2019 about the project! I couldn’t attend it, but I watched the livestream. :)

It was my first year as a mentor. It did feel a little weird, since I never participated in GSoC as a student, and I’m also a student myself. Nevertheless, the project was finished successfully, so we have a shiny new feature. Most of the work is already merged and will be available in GNOME Games 3.34.0. The two remaining changes will be landed early in 3.36.x cycle.

Andrei is awesome to work with. He started coding early in community bonding period, and needed next to no handholding, so for the the most part of the coding period my involvement was simply answering questions and reviewing code, later testing and reporting bugs.

While the project was finished, I think there are things I think I could do a lot better as a mentor:

While we were focused on the technical side of the project, I didn’t pay enough attention to the social side. Namely, I didn’t track GUADEC announcements closely (since I couldn’t attend it this year anyway), so Andrei almost ended up missing it. Thanks to Gaurav Agrawal who helped Andrei with registration and requesting travel sponsorship.
Until the last few weeks, the project consisted of a huge merge request which quickly got unwieldy. For planning we used Matrix. I saw other people using more sophisticated workflows for GSoC, such as networks of small merge requests and issues in interns’ forks on GitLab, which can be reviewed and merged one by one, I think we should have used a workflow like that as well.
I ended up postponing 3.33.90 release for a week, merging many followup changes during the freeze, and even then merging changes up until 3.33.91 (Since Games is not a core app, we are allowed to do that, but usually we try to follow the freezes nevertheless). This probably wasn’t a good thing to do and I apologize to the translation teams. :(

All in all, I want to thank Andrei for the amazing contribution and for being an awesome person in general. :)

1. Transitions and visible-child