Login and unlock in GNOME Shell 3.36

The upcoming GNOME 3.36 release includes a major update to the system login and unlock experience. The new design has been anticipated for a long time, and we’re excited that it has finally arrived!

Some history

The lock screen in GNOME 3.6

GNOME’s existing login and unlock design has been largely unaltered since it was first introduced in GNOME 3.6, back in September 2012. That’s seven and a half years ago! It’s therefore no surprise that we’ve wanted to update the design for some time.

The initial round of design work for the new lock screen took place in 2017, at the GNOME UX hackfest in London. There, the GNOME design team, along with GNOME Shell developers, reviewed the goals and requirements, as well as the issues with the existing design, including the main areas of feedback that we’ve had.

Lock screen planning at the 2017 UX hackfest

The design concept that we came up with during that event has undergone significant refinement since 2017, but the basic principles and goals remain the same. These can be summarised in three words: smooth, beautiful and personal.

Reducing friction

Unlocking the system is something that people do over and over, so it’s important that it doesn’t feel frustrating, and it needs to get you from A to B as quickly as possible. One of the goals of the design update was therefore to reduce the amount of friction involved in unlocking (or logging in for that matter).

From this perspective, the most obvious change in the lock screen is that the grey login screen is now gone. Rather than removing the “shield” to show the password field, the password field is shown right on the lock screen itself. This doesn’t necessarily change how many keystrokes or clicks are required to unlock, but it does reduce how many distinct visual steps are involved, and makes the whole experience feel faster and more direct.

We’ve also incorporated other changes which are intended to make it quicker and easier to unlock. There are more methods to show the password field than before: you can click with a mouse or touchpad button. You can also scroll with a mouse, or swipe with a touchpad or touchscreen. Small changes like this add an extra degree of convenience in many situations.

Other changes in this area are planned for the future. One of the biggest is to jump straight to the last-logged in user after boot, rather than requiring a user account to be selected every time. In most situations, this will take an extra step out of logging in: all you’ll have to do is boot, then enter your password, and off you’ll go.

More happy times

If getting to your work more easily is one of the big themes for the updated design, then the other is having a more joyful experience along the way. Unlocking is a chore, but it shouldn’t have to feel like one. It can and should be pleasurable.

With the new design, we wanted unlocking to be beautiful, and we’ve put a lot of work into visuals. The most obvious change is the use of the blurred background, which we love. Not only does it look great, but it also serves as an effective background for legible, light-weight typography, so it’s possible to have elegant text, without the need for heavy type or drop-shadows.

There’s also been lots attention to detail throughout design and development process, which we hope will make the login and unlock experience feel delightful to use. This includes things like the updated layout of the authentication elements, the subtle use of animation to communicate incorrect authentication attempts, and the transition from the clock to the password field.

Making it personal

A final, and perhaps more subtle, aspect of the updated design is the effort to make it feel personal. We want to connect the login and unlock with the user’s desktop experience, so the user gets a hint of their session even when they are logged out or the system is locked. The main way we’re doing this is the use of the blurred wallpaper, which is intended to communicate what lies on the other side of login/unlock.

Mockup for the redesigned user selection screen

The use of strong personal elements, like the user avatar and blurred background image, is also something that we intend to build on in future versions, by bringing them into the user selection screen itself. The goal here is to bring user selection alive: to make it more expressive and indicative of the person to which each account belongs.

Getting to this point

On the development side, the login/unlock redesign has primarily been carried forward by Georges Stavracas, with significant assistance from Umang Jain and Florian Müllner. It has been a significant amount of work and we owe them a lot of gratitude for the time and effort they have invested in this project.

Next steps

The login/unlock redesign was merged slightly late in the GNOME development cycle. To compensate for this, it is undergoing an intensive testing and bug fixing effort. Help is needed with this, and there is a test plan available for those who would like to try the new design and report any issues they find.

As mentioned above, looking beyond GNOME 3.36, there are a number of aspects of the design which are yet to be implemented and which we hope to have for the next release, version 3.38, so there is hopefully plenty more to look forward to in this area.

This Month in Mutter & GNOME Shell | December 2019 & January 2020

GNOME Shell

Unified Layout for Dialogs

One area of focus during this cycle was unifying the layout, content structure, and feel of dialogs in GNOME Shell. Many dialogs were redesigned, polished, and updated as a result of this effort:

Improved Gesture Support

The gestures to switch workspaces, and navigate through the application grid pages, were dramatically improved, and now follow the touchpad movement.

Password Peek

The password entries in GNOME Shell now support peeking passwords! Take a look:

Folders as Dialogs

A major change on how GNOME Shell handles application folders has landed this month. Instead of being displayed as popups within the application grid, folders are now displayed above it in the form of a dialog.

Miscellaneous

A new blur effect was added to GNOME Shell. It will be used by future work, such as a redesigned lock screen, among others.

Due to the VP9 encoder being too CPU-intensive, the built-in screen recorder changed back to the VP8 encoder.

As part of a code refactoring effort, the bits of StBoxLayout that deal with hiding overflowing content were moved to a new class, StViewport. This simplifies in various ways the implementation of overflowed content in GNOME Shell.

When managing the icons in the application grid, such as creating, renaming, or deleting folders, or moving app icons in and out of these folders, the icons will animate to their positions smoothly now.

GNOME Shell’s performance profiling framework was fixed and updated to also work on Wayland.

The SCSS code that is used to generate the default GNOME Shell theme was completely revamped and reorganized, making it easier to find specific theme classes. It also received a small visual update.

At last, GNOME Shell now supports NVidia GPU offloading by showing the “Launch on Discrete GPU” entry on systems with a NVidia discrete GPU available.

Mutter

More Cogl Cleanups

Cogl is the GPU rendering abstraction used by Clutter to render its contents on screen. Clutter itself is the base of Mutter and GNOME Shell, which add window management and compositing on top of Clutter.

During the past few months, there were continued efforts to cleanup and modernize Cogl by removing dead and unused code, using more modern OpenGL features, etc.

This work continued during December and January too, with more dead code saying goodbye, and more modern code being welcomed to the codebase.

Implicit Cogl API Removal

Due to historical reasons, Cogl has two major APIs: an explicit one, with properly defined types, and functions that receive the objects they should act on; and an implicit one, that is closer to the OpenGL model, and acts on an implicit state machine with the objects to be worked on.

Many parts of the Clutter, Mutter, and GNOME Shell codebases were using the implicit Cogl API. Those usages were updated, and allowed us to remove most of the implicit API from Cogl.

ClutterOffscreenEffect Improvements

ClutterOffscreenEffect is an effect applied on UI elements that allow them to be rendered in offscreen framebuffers. This allows interesting, shader-based effects to be applied to these elements, such as contrast and brightness, blur, among others.

A few bugs were found in this effect, and were ironed out. It also received some optimizations that allow the effect to release GPU memory more often.

Introduction of ClutterSeat

Clutter’s representation of input devices was designed in the X11/XI2 times, and the API was modeled after it. This change brings the internal input model closer to Wayland.

While it may sound an unimpressive change, it’s the cornerstone for an unification of the multiple input grabbing mechanisms in use by GNOME Shell.

Between other internal refactors that are now possible, it will also facilitate the introduction of an input thread in the native backend, so the pointer cursor is no longer frozen on stalls.

This will be covered in detail in a separate article.

Improve layout phase of no-layout actors

The Clutter toolkit follows the traditional rendering pipeline, composed of the layout, paint, and pick phases. The layout phase is where the elements (in Clutter terminology, “actors”) get to know their position and size; the paint phase is where these actors paint their contents on the framebuffer; and the pick phase is where Clutter figures out which actor is hovered by the pointer.

When we know beforehand that an actor is not going to move around the screen, we can tell Clutter about it, and the layout phase can be optimized to avoid some calculations. This mode is called “no-layout”.

During January, Clutter received another set of optimizations when dealing with no-layout actors. This can reduce CPU use on some specific scenarios, such as when dragging windows.

Miscellaneous

The ClutterContent interface now is better integrated in the size negotiation phase of ClutterActors, by making actors that follow their content’s sizes report their content’s geometry.

Many improvements to how Mutter tracks Wayland windows landed. Specifically, a crasher related to wl_subsurface was ironed out, a unit test framework was introduced, and a harmless runtime warning was fixed.

When dealing with multi-GPU setups, Mutter needs to share framebuffers across GPUs. There are currently three ways to do that:

  1. The secondary GPU does an accelerated copy of the primary GPU framebuffer;
  2. The primary GPU does an accelerated copy of its own contents to the secondary GPU framebuffer;
  3. A slow, non-accelerated framebuffer is created on the secondary GPU, mapped to CPU memory, and the primary GPU contents are copied to this memory;

Mutter now features an additional way to share framebuffers: the primary GPU exports a framebuffer that is imported and read directly by the secondary GPU. This method is useful with virtual secondary GPUs, such as DisplayLink docks.

This Month in Mutter & GNOME Shell | November 2019

GNOME Shell

GNOME Shell saw many improvements during November. The commit log was dominated by cleanups, but a few improvements and polishments also found their way into the code.

The authentication dialog received a batch of bugfixes, many cleanups of deprecated objects and functions landed. The top panel’s application name is now correctly sized by hiding the spinner near it.

GNOME Shell’s cache of icons and textures received a fix to invalidate properly when dealing with scaling changes. All-day events are properly displayed in the messaging menu now.

Finally, the Alt-Tab switcher now doesn’t mistakenly show an overflow indicator when the list of windows fits the screen size.

Libcroco Removal

The libcroco dependency was dropped by importing the source files into St. This is an important step in getting rid of libcroco, which is a dated CSS parsing library.

App Grid Improvements

The icon grid saw an important fix to dragging application icons. The icons were not properly being destroyed, and thus were piling up after dragging and dropping them over time. This fix was further improved to work on more situations. This set of fixes was backported to the 3.34 release.

A nice visual improvement landed on the page indicator of the icon grid.

 

System Font

GNOME Shell now respects the system font!

Mutter

For Mutter, November highlights were the introduction of regional clipping in Cogl, and big code cleanups.

Regional Clipping

When applications and GNOME Shell draw themselves, they communicate which parts of their contents changed. That information allows Mutter to submit only the changed contents to the monitor, which is an important optimization.

Example of GNOME Clocks being partially redrawn
Example of GNOME Clocks being partially redrawn. The changed parts are painted in red.

Until GNOME 3.34, Mutter would calculate the bounding rectangle between all the regions that changed:

Mutter would submit the bounding box of all updated regions (in blue). In many situations, such as the above, that would include more than necessary.

This month, Mutter received the ability to update multiple regions independently, without using the bounding rectangle. In the example, Mutter now updates only what has actually changed:

The regions that Mutter submits (in blue) now matches the regions that really changed in the first picture (in red)

This yielded a significant improvement too! Under some circumstances, this change alone can reduce the time to submit frames by up to 44%.

Shadow Buffer

In some situations, in the native backend we now use a shadow buffer to render the stage off-screen before copying the content over to the actual buffer handed over to the display panel. While this may sound counter productive, it significantly increases performance from unusable to fairly pleasant on those systems that need it.

Other Highlights

We now prevent full window redraws when using dma-buf or EGLImage buffers on Wayland (mutter!948). This fixes partial updates of windows on Wayland, which can reduce the amount of data transferred between GPUs, CPUs, and the monitor. Together with the regional clipping explained above, this should significantly help saving battery.

Many, many Clutter and Cogl cleanups (mutter!921, mutter!819, mutter!933, mutter!932) landed too. These merge requests remove deprecated functions and features that, as time passes, are an increasingly burden for maintenance, and in some cases also prevent improvements and optimizations . About 28000 lines of legacy code has been cleaned out from Mutters own Cogl and Clutter versions so far, since we entered the 3.36 development phase. Extension authors, please make sure your extensions don’t use any of the removed code.

One legacy feature that dates back to when Clutter was a separate library used to write client applications was removed (mutter!911) from Mutter’s internal copy of Clutter. Not clearing the stage doesn’t make sense on a compositor.

Xwayland games that run fullscreen and change resolution should behave better now (mutter!739).

We’ve also seen a few bug fixes landing, for example fixes to Drag n’ Drop, a couple of memory leak fixes, crash fixes including one related to hot plugging and another that sometimes occurred when running Intellij, and a bug fix avoiding stuck full screen content.

A Review of GNOME Shell & Mutter 3.34

The last GNOME release, named “Thessaloniki”, was busy for GNOME Shell and Mutter. Many changes, ranging from code cleanups to architectural changes to performance improvements to new features landed.

Let’s take a look at the major highlights for the GNOME 3.34 release.

GNOME Shell

JavaScript Updates

GNOME Shell uses GJS, the GNOME JavaScript engine, to run. With the latest updates to GJS, such as the JS60 migration, GNOME Shell saw important updates making the codebase use modern tools and code practices.

Implicit Animations

One of the most important improvements that were part of GNOME Shell 3.34 was the transition to Clutter implicit animations. So far, GNOME Shell has been using the Tweener framework, which is written completely in JavaScript and, as a consequence, required extra work to communicate the animations to Clutter.

Using the implicit animations framework provided by Clutter allows GNOME Shell to skip a lot of JavaScript → C jumps, and thus, take less resources to run animations.

CI Integration

Testing GNOME Shell with CI is slightly tricky due to it and Mutter always being in lockstep. It is common that GNOME Shell from the git master branch needs Mutter from git master as well. Finding a good way to handle that prevented CI from landing earlier, but during this cycle we crafted custom container images and wrote the necessary scripts to allow for that.

Now, when running CI, GNOME Shell is tested with Mutter in tandem. Furthermore, on Mutter side, we’ve added a CI step to build and test GNOME Shell as well, catching many errors that would otherwise take some time to be found.

New Extensions Tool

GNOME Shell 3.34 also ships a new gnome-extensions tool, making it easier to create and manage GNOME Shell extensions. Due to the automation this new tool provides, such as extension templates, bash completion, packing, and testing extensions, it should be a significant improvement for extension authors.

Accessibility

With the ongoing transition to Wayland, GNOME Shell is taking ownership of many features that required random applications have full access to windows and their contents. One area where this bad practice is recurrent is the accessibility stack.

With GNOME Shell 3.34, a big coordinated effort was put in filling the missing accessibility bits  when running a Wayland session, such as Locate Pointer, Click Assist, among others.

Folder Management

Last but not least, in GNOME Shell 3.34, it is possible to create, rename, and delete folders using Drag n’ Drop actions.

GNOME Shell 3.34 in Numbers

GNOME Shell 3.34 had 309 files changed, with 50,022 lines added and 26,924 removed. 59 merge requests were merged, and 1 was closed. 71 issues were closed by commits.

Developers with the most changesets:

| 289  | Florian Müllner               |
| 84   | Marco Trevisan (Treviño)      |
| 65   | Jonas Dreßler                 |
| 32   | Georges Basile Stavracas Neto |
| 19   | Carlos Garnacho               |
| 11   | Ray Strode                    |
| 10   | Ryuta Fujii                   |
| 9    | Kukuh Syafaat                 |
| 7    | Cosimo Cecchi                 |
| 7    | Jordi Mas                     |

Mutter

Sysprof

Sysprof is GNOME’s profiling framework, built on top of the `perf` API. With the GNOME 3.34 release, Mutter now integrates with Sysprof and is able to profile a lot of its internals.

Profiling Mutter is an important requirement to detect potential bottlenecks in the algorithms and the codebase, and have a proper understanding of how, where, and why things are the way they are.

GPU Hotplug

Mutter is now able to detect and use GPUs, usually external, that are plugged while the GNOME session is running.

KMS Transactions

In order for Mutter to support more complex hardware capabilities it needs to know how to utilize the atomic KMS kernel API. A first step in this direction was taken this cycle by implementing an internal transactional KMS API. Right now this API is backed by a non-atomic implementation, in order to make sure that we don’t break anywhere we currently fully function on, but in the future having this API will enable us to have mutter utilize more advanced hardware capabilities such as overlay planes and framebuffer modifiers. It is also a big step closer to being able to isolate KMS interaction into a dedicated thread eventually enabling low latency input device feedback once we move input processing into its own dedicated thread as well.

Improved Frame Scheduler

The algorithm that Mutter uses to schedule when to update the contents of the
monitor was fine-tuned and is now able to deliver new frames more smoothly. The new algorithm gives applications more time to draw their contents and notify Mutter about it. As a consequence, Mutter is much more likely to update frames in time for the monitor to display.

NVIDIA on Xorg frame throttling changes

Further in the past, Mutter used glxSwapBuffers() to throttle frame drawing when using the proprietary NVIDIA driver. This caused issues, as it’d block the main thread for long period of times if there was constantly new frames being scheduled. To mitigate this, Mutter developers introduced functionality imitating the GLX_INTEL_swap_event extension used by the Intel driver to make it possible for swapping buffers asynchronously, by receiving a “completion event” (swap event) later on.

This was done using NVIDIA specific GLX extensions combined with running a separate thread, where the “swap event” was generated, in contrast to when using GLX_INTEL_swap_event, where the Intel driver itself did this.

In practice, this caused issues, as it the relevant GLX extension implementation in the NVIDIA driver tended to be CPU heavy. However, due to the changes this cycle to our frame scheduling algorithm, the initial problem is now mitigated by scheduling frames in a way that we will we avoid blocking on glxSwapBuffers() for as long, meaning we could remove the CPU heavy “swap event” imitation code. This should mean less CPU usage when using the NVIDIA driver and displaying an application that continuously renders frames.

Graphene

Graphene is a library implementing complicated math functionality related to vertices, matrices, rectangles and other types usually related to 3D programming. In order to decrease maintenance burden on Mutter, which had its own set of complicated math functionality doing practically the same thing, we went and replaced most of that code, and switched to using Graphene directly. Not only did we decrease the amount of code we have to maintain, we also got all the fancy new improvements done to Graphene that we would otherwise be without!

For GNOME 3.34, all but matrices were converted to the corresponding Graphene types. That’s because Cogl and Graphene matrices have different semantics and, thus, will require more scrutiny.

XWayland on Demand

Mutter and GNOME Shell 3.34 earned the ability to run as a pure Wayland compositor, while still providing seamless compatibility with X11 by starting XWayland when required by applications. This is the culmination of much prior effort at refactoring internals and ironing out indirect X11 dependencies throughout Mutter and GNOME Shell specifically, and the GNOME session generally.

Being able to start Xwayland and other related X11 services on demand for legacy applications came out organically, and demonstrates we are no longer tied by our legacy support.

This feature is disabled by default, and can be enabled by adding ‘autostart-xwayland’ to `org.gnome.mutter experimental-features`.

Real-Time Scheduler

When running as a Wayland compositor and display server, it is important that Mutter keeps its reponsiveness all the time. However, Mutter runs as a regular user application, and is subject to starvation in various scenarios, such as intensive I/O.

Mutter 3.34 now requests real-time priority, which gives it more priority than
regular applications. This avoids stalls and stutterings when running GNOME Shell as a Wayland compositor.

This feature is disabled by default, and can be enabled by adding ‘rt-scheduler’
to `org.gnome.mutter experimental-features`.

MetaShapedTexture as a ClutterContent

MetaShapedTexture represents the visual contents of an application’s surface. It takes care of making sure the content is cropped and scaled properly, and tries to optimize drawing depending on what is opaque, and what is not.

ClutterContent, on the other hand, is Clutter’s way of defining deferred rendering instructions, also known as render nodes. This enables more efficient batching of draw calls, compared to the implicit batching implemented by the Cogl journal. This cycle we took the MetaShapedTexture, which was a ClutterActor subclass manually issuing Cogl drawing instructions, and turned into a set of fully deferred rendering instructions as defined by ClutterContent.

This not only improved batching of draw calls but greatly decreased the size of the Clutter actor tree, giving us a significant performance boost.

Here’s a quick comparison:

Graph showing that MetaShapedTexture as ClutterContent reduces frame times when many windows are open.
Turning MetaShapedTexture into a ClutterContent implementation (green bars) reduces frame times when many windows are open, compared to the previous implementation (yellow bars). Courtesy of Jonas Dreßler.

 

Mutter 3.34 in Numbers

Mutter 3.34 had 766 files changed, with 34,249 lines added and 37,268 removed. 74 issues were closed by commits.

| 122 | Jonas Ådahl                   |
| 109 | Carlos Garnacho               |
| 82  | Marco Trevisan (Treviño)      |
| 52  | Olivier Fourdan               |
| 48  | Georges Basile Stavracas Neto |
| 40  | Florian Müllner               |
| 30  | Adam Jackson                  |
| 27  | Daniel van Vugt               |
| 24  | Hans de Goede                 |
| 24  | Robert Mader                  |
| 19  | Jonas Dreßler                 |
| 16  | Niels De Graef                |
| 16  | Pekka Paalanen                |

Every shell has a story

… a wise someone once muttered while walking on a beach, as they picked up a shell lying on the sand. Indeed, every shell began somewhere, crossed a unique path with different goals and driven by different motivations. Some shells were created to optimize for mobility; some, for lightness; some, for speed; some were created to just fit whoever is using it and do their jobs efficiently. It’s statistically close to impossible to not find a suitable shell, one could argue.

So, is this a blog about muttered shell wisdom?

In some way, it actually is. It is, indeed, about Shell, and about Mutter. And even though “wisdom” is perhaps a bit of an overstatement, it is expected that whoever reads this blog doesn’t leave it less wise, so the word applies to a certain degree. Evidently, the Shell in question is composed of bits and bytes; its protection is more about the complexities of a kernel and command lines than sea predators, and the Mutter is actually more about compositing the desktop than barely audible uttering.

What Now?

For a long time, the development of Mutter and GNOME Shell was surrounded by silence. This blog is a humble attempt to bring those two critical components of the GNOME desktop to the spotlight, even if only a tiny bit more.

It would be naive to say that posts will be published in a consistent frequency, but the initial goal is monthly development reports and eventual one-shot deep dives into various bits and pieces of these two components.

So fasten your seatbelts, and enjoy the ride.