The volunteers and contributors working on Mutter and GNOME Shell have been busy in the past couple of months — so much so that we didn’t have bandwidth to write the May development report!
As a consequence, this development summary will have an above average number of changes to highlight.
GNOME Shell
Preparations for Customizable App Grid
As part of the preparations for a customizable application grid, a new layout manager was written and replaced the current icon grid code. This new layout manager is in many ways more suitable for current and future changes:
It follows the delegation pattern that is common to Clutter. As such, it is a layout manager, and not an UI element itself.
It allows more precise control over the how the grid is displayed.
It uses modern JavaScript practices and is, in general, a more maintainable and comprehensive code.
The most visible impact is that it now selects a row x column configuration that is closest to the aspect ratio of the display:
New layout manager on portrait mode
There are still improvements to make, especially with ultra-wide displays, but the foundation work is already there, and it will be vastly easier to fine-tune the behavior of the app grid on different scenarios.
This new inspector has been useful for developing GNOME Shell, and hopefully it’ll help extension developers too.
App Folder Dialog Updates
App folder dialogs received a bunch of visual and behavioral improvements, such as covering the entire monitor, and not changing the size of the app grid itself. Take a look:
These dialogs are now paginated, and fixed to 9 app icons per page:
Paginated folder dialogs with 9 items
Like the app grid, folder dialogs now also have better support for touchpad gestures and Drag n’ Drop.
Updates to the Message List Popup
This year, GNOME Shell has a Google Summer of Code intern working on the messages dialog. As a preparation for this project, some cleanups and reorganizations of the message list popup landed. More work in this front is happening, and an influx of improvements is expected to come soon, stay tuned!
An unfortunate oversight was causing the Do Not Disturb setting to be reset on startup. This bug was fixed. A potential D-Bus race condition when creating MPRIS media players was corrected. App icons do not vertically stretch in the top bar anymore. These bugfixes were backported to GNOME 3.36.
The code that deals with workspaces in GNOME Shell is old, but a large number of cleanups to it has landed (!1119, !1251, !1294, !1297, !1298, !1307, !1310, !1313, !1320, !1333), and even more is under review. These cleanups were much needed in order to improve the overall quality and maintainability of the codebase.
The Extensions app saw some improvements too. The Logout button now workscorrectly.
When the host system changes timezones, GNOME Shell now properly updates the timezone offsets of the “World Clocks” section of the messages popover.
While routinely profiling Mutter, it was noticed that an abnormally high number of safety type checks were happening in a rendering hot path, during the redraw cycle. Those checks were removed.
Combined, these changes are of notable significance due to how expensive it is to recalculate the layout of actors. Some of them are also required for per-CRTC frame clocks.
Rendering Pipeline Improvements
Cogl now supports setting a maximum mipmap level, in addition to the minimum one, and background set a maximum mipmap level. This avoids creating mipmaps that won’t be used.
Last year, MetaShapedTexture was made into a ClutterContent implementation. This change was important for a multitude of reasons, and will play a special role in the future with upcoming cleanups. However, it also introduced an unforeseen regression: Clutter paints ClutterContents before running the main painting routines, and this broke the existing culling mechanism of Mutter. After some investigation, culling wasfixedagain.
Mutter also received a seriesofimprovementstoitstestsuit. These improvements range from fixing broken tests, make CI more reliable, add more tests, reorganize the entire test suit, among other changes.
This is the second of a series of posts on the design of GNOME Shell, as part of an ongoing initiative to improve the core shell experience. In the previous post of the series, Allan gave a general introduction to that initiative. This post is meant to provide some more context and touch on longer-term ideas we’re thinking about. It’s worth noting that many of these are still at an early stage, and subject to change.
It’s hard to believe, but we’re coming up on 10 years of GNOME Shell. Though GNOME 3.0 wasn’t released until April of 2011, design and development were in full swing in mid-2010, and had been for some time.
Of course, the shell has not been standing still since then. Many important areas have been refined and iterated on since those early days, including the notifications, the app grid, and the system status area.
A lot has changed since 3.0. Did you remember the search bar used to be on the right? (Screenshot from Wikimedia Commons)
However, other parts of the interface haven’t changed much since the early days, and we know that some aspects of these old designs could be improved. There are a number of core elements of the shell that we’ve wanted to refresh for some time now, and as the previous blog post discussed we’re hoping to move forward with some of it in the near future.
Given that background, this post is meant to provide an overview of where we see the strengths and weaknesses of the existing design, as well as areas that we’re interested in improving. This includes the areas highlighted in Allan’s post, but also expands the list to cover other initiatives that are either ongoing, or which we would like to see happen.
Strong Points
Distraction Free
One of the core ideas behind the original design of the shell was getting out of your way, and reducing distractions as much as possible. While you’re using apps there are no colorful icons, applets, or other clutter on the screen: It’s just you and the thing you’re working on.
Long before big tech started talking about giving people back control over their attention, “digital wellbeing”, and “time well spent”, GNOME Shell was built around helping people deal with distractions and focus on what really matters. Needless to say, this is more relevant than ever, and something we want to continue to focus on.
If you’re curious, here’s some further reading on the thinking behind the initial 3.0 shell design:
The GNOME Shell overview is a single place to go for app launching and switching, window management, and system-wide search. It’s the center of the shell experience, the place to see what’s happening on the system at a glance, switch activities, or start new ones.
The overview in 3.36
No other desktop OS has a core workflow this simple and straightforward, and many people really really love it. This is another core strength we want to build on.
Just Search
Most operating systems have some kind of system-wide unified search nowadays, but the GNOME Shell one is especially great because it’s seamlessly integrated with the “single entry point” workflow. This means there’s no extra button to click, or separate keyboard shortcut to learn. Just hit Super and type.
Search results in 3.36
Switching > Launching
Most of the time when leaving the current activity you don’t start a new one (e.g. by opening a new app), but switch to another running activity. GNOME Shell optimizes for this by putting multitasking front and center in the overview, ahead of launching new apps.
Ergonomics
A key reason for the success of the “single entry point” workflow is how easy it is to open the overview across input devices (most of them anyway, but more on that later).
With a mouse you just flick your cursor up to the top left corner. This is very ergonomic because you don’t have to aim, you just move the mouse in that general direction.
If your hands are on the keyboard you can hit a single key, conveniently located in the bottom left of the keyboard, to enter the overview.
These methods quickly become second nature, so much so that people report feeling a visceral lack of control when using other systems. Windows and macOS have equivalents to many of GNOME Shell’s individual features, but there’s no way to do it all from one place in such an ergonomic way.
Areas for Improvement
While there’s a lot to like about the status quo, we’re aware that there are also a number of issues with it, both structurally and usability-wise, and there are a lot of things that work well today, but we think could be even better.
Launching & Initial State
One of the most visible shortcomings of the current shell layout is the bootup experience. After login you’re greeted by an empty desktop, and when you open the overview it’s also empty, since there are no open windows. Though this is not something you encounter very often, it’s definitely not a good first impression.
Overview empty state in 3.36
There are also issues with launching apps from the app grid more generally: It’s not very prominent or easy to get to, and it’s split into two separate views, each with its own problems. Apps don’t have fixed positions in either view, because the All view is alphabetical, so things move around when you install or remove apps, and the Frequent view constantly changes depending on usage.
In improving the app grid, we’d like to move towards something that allows people to develop spatial memory for where certain apps are, and organize them in a semantic way.
Multitasking Workflow
Our current multitasking model consists of two levels: workspaces, which are laid out next to each other in a vertical row, and windows, which are stacked behind one another. This two-level system can be quite powerful, but there’s no automatic onramp to the workflow. By default, if you launch apps they all just pile up on the same workspace.
It’s possible to use GNOME Shell with just a single workspace, but it can be awkward, e.g when there are too many windows for the overview to be an effective way of switching because the thumbnails are so small. All windows stacking up behind one another also makes it hard to do things that involve multiple windows at the same time, such as keeping two windows side by side.
The overview with many windows on a single workspace
People often mention that once they were shown the workspace feature (e.g. by a friend or coworker), it “clicked” for them and they started using and loving it. Of course, this kind of discovery doesn’t scale, and there’s clearly a gap in the learning curve.
This issue isn’t new for workspaces (across all implementations, not just ours): You need to know about the feature, actively decide to use it, and put in some manual work. In our case this is made worse by the fact that the system isn’t able to “remember” open windows and workspaces across reboots.
Ideally we’d like to make multitasking and window management feel more organic than the current two-level system, and reduce the amount of work people have to do to keep things organized and easy to navigate.
Gestures & Spatial Model
Some parts of GNOME Shell have a clear, simple spatial model which makes navigation feel effortless. For example, workspaces are arranged in a vertical row, which you can cycle through via touchpad gestures or directional keyboard shortcuts.
However, this isn’t the case for everything, which is why parts of the shell can feel disconnected and difficult to navigate. A good example of this is the app grid: The first page opens with a “swarm” animation from the app grid icon, but then additional pages appear from outside the viewport.
Ideally the shell should have a clear, unified spatial model, so you always instinctively know where things are (even when they’re off-screen), and how to get to them, across input methods and form factors.
In order to enable that we need a more comprehensive set of gestures for navigating the core parts of the shell, as well as more intuitive and discoverable keyboard shortcuts. The same thing largely applies to touchscreens as well. Some gestures there are quite nice, but others are not “stick-to-finger”, or physically difficult to perform (e.g. pinch to open the overview).
One of the nice things about having a simpler spatial model and directional gestures to navigate the shell is that it also helps keyboard navigation. This is because it allows for directional shortcuts (e.g. Super + PgUp to move to the workspace above) which are more intuitive than non-directional ones (e.g. Super + M to open the notification popover).
Notifications & Agenda
The calendar popover contains a lot of different features (notifications, calendar, world clocks, weather). This can result in it feeling a bit large and unwieldy. Seeing as not everyone uses all the features in the popover we’re interested in giving people a way to customize which ones they want.
Ideally we’d like the features in this popover to be grouped in a more semantic way and easier to navigate to, especially with touchpads and touchscreens.
Notifications & Agenda popover in 3.36
In addition to structural changes, we also want to make the notifications themselves better, for example by making notification actions accessible in the popover and grouping notifications by app. Mariana Pícolo is working on some of this as part of this year’s GSoC.
System Status Area
The system status area works well for the most part, but there are some parts we’d like to improve. For example, taking actions on individual items (e.g. toggling Bluetooth) always requires expanding the item’s sub-menu. Other items (e.g. Night Light) are only available in the menu when they’re enabled, which can feel unpredictable.
System status popover in 3.36
Search
Our current search works well overall, but there are still areas where we think it could be better.
For the most part, search is currently just a portal to things inside apps (e.g. files, settings panels, contacts). However, we’ve found that search is often most interesting when it gives you something directly–launching an app, performing a calculation, looking up the weather, etc.
Having more such integrations and richer ways to interact with content from search is something we’re interested in exploring, along with generally expanding the kinds of things you can search for, e.g. your currently open windows.
Goals
Based on the above analysis, here are some tentative high-level goals for where we want to take the shell over the next few years:
Distraction free
Single entry point
Feels quick & easy to navigate
Structurally and visually elegant
Self-teaching
In order to achieve these goals, these are some more concrete things we want to focus over the next few cycles (the previous blog post discusses these plans in more detail):
Better onboarding and bootup experience
Easier app launching
Simpler, unified spatial model
Better touchpad and touchscreen gestures
Richer notifications
More medium-term, we’d also like to look at these:
More organic workspace workflows
Streamlined window management
Remember open apps across reboots
Improved system status area and search
Of course, some of these goals are much easier to achieve than others. Some of them may take years of work and multiple iterations to get right. That said, we think it’s important not to think about these ideas in isolation, but as part of a more comprehensive vision for the future.
Next steps
The previous blog post lays out a fairly clear roadmap for the first few things we want to address. As that work becomes more concrete, we look forward to sharing and discussing it with the wider community. We hope this post will help provide context to that discussion.
These are exciting times, and we can’t wait to share more. Stay tuned!
A bit later than usual, but nonetheless here the changes that happened during April on GNOME Shell and Mutter.
GNOME Shell
The command-line extensions tool received a round of improvements, and now reports extension errors better. Switching the scale of a monitor now should update all interface elements properly on GNOME Shell. Another quality of life improvement that landed was the inclusion of ASCII alternatives in the search index, which for example allows “eteindre” match “éteindre” (French for “power off”).
GNOME Shell now integrates with the parental controls technology being developed across the GNOME stack. If there are user restrictions in place, GNOME Shell now filters the applications that are not supposed to be used by the particular user.
One important improvement that landed during April is the rewrite of GNOME Shell’s calendar daemon. This updated version should prevent a lot of heavy background processing of events. Given the extents of the improvements, this is being considered for backporting to GNOME 3.36, but the size of the changes are also considerable. Testing and validation would be appreciated.
April then ended with the release of both GNOME Shell 3.36.2 as well as 3.37.1.
The most prominent addition to Mutter during April was the introduction of Wayland fullscreen unredirect. This code has been under review for many months, and required coordination between different parts of the stack, to work properly. Unfortunately, because it requires a very recent version of Xwayland (1.20.8) containing the fixes necessary for it to work properly, it is not suitable for backporting.
Improvements to the screencasting code landed in preparation for further improvements to GNOME Shell’s built-in screen recorder. We hope to be able to have a single code path for capturing the screen contents, regardless of whether the consumer is the Desktop portal, or the built-in recorder.
Also an issue many users had ran into where the Super key did not work as it should when using multiple keyboard layouts in the X11 session was fixed!. A handful of other bug fixes and improvements was made for the GNOME 3.36 stable branch was also included in the 3.36.2 release in the end of the month.
Like GNOME Shell, April ended with Mutter’s 3.37.1 release as well.
With GNOME 3.36 out the door, it’s time to start thinking about what comes next. Those of us who work on GNOME UX are really proud of the latest GNOME shell release. It includes some major updates, particular the new lock screen and updated visuals, as well as a host of smaller improvements. 3.36 feels really nice to use, and has got a great response.
The new lock screen in GNOME 3.36
The lock screen work that we landed in 3.36 was the outcome of a long-running programme of UX work, which we first put together at the GNOME UX hackfest in London, back in 2017. There are still some outstanding pieces of the login/unlock experience that need to be filled in, and this is something that we hope to work on over the coming development cycle. However, we are also turning our attention to other aspects of the shell, which we have wanted to update for some time.
In the rest of this post, I’ll describe some of the areas that we’re hoping to improve, before going on to talk about how we’re going to do it.
Focus areas
The areas that we are looking at mainly relate to the overview, although in places they touch other areas of the experience. They are:
Application launching
One thing we’ve wanted to improve for some time is the reliance on the alphabetical grid of launchers. This alphabetical organisation is mostly useful for finding applications by name, rather than presenting the apps that are most important or most likely to be used. This is something that the grid of frequent apps attempts to compensate for, but the overall experience doesn’t fit together as well as we’d like.
The application grid
For application launching, the main goal is to make the apps space more useful – it needs to be easier to find the app that you want to launch. However, we also want to make it a more engaging and personal part of the experience, so the apps space feels like it belongs to you.
Overview spatial organization
We are all familiar with the organisation of the overview: window thumbnails in the middle, dash on the left, search up top, workspaces on the right. Functionally this works fairly well. However, the layout could be better in terms of how the pieces fit together. In particular, the arrangement of the different elements could do a better job of communicating how they relate to one another, especially in terms of navigation.
The Activities Overview
Boot and empty states
Boot into GNOME 3 and you’re presented with an empty desktop. Open the Activities Overview, and you’re again presented with an empty space in the middle of the screen. This experience isn’t particularly helpful, and doesn’t draw or guide the user into the experience.
Overview initial state
We’ve been aware of these disadvantages of the current design for almost the entire history of GNOME 3, and we have experimented with a few different solutions, but never managed to get them to a usable state. Now, as we take another look at how the overview is arranged, we’d like to have another attempt at getting this right.
Touchpad navigation
Right now, the touchpad gestures to move between workspaces are fairly straightforward: 4 finger swipe up and down. However, we currently don’t have an easy gesture to go in and out of the overview and, once you’re in it, we don’t have an easy way to navigate between the different areas (app launching and search in particular).
Being able to scoot around different areas of the desktop using a touchpad (or, indeed, a touchscreen) would be a big win and is something we are keen to allow. In order to do this, we need a simple model that people can use to navigate around, rather than having to learn multiple sets of gestures.
How we’re going to do it
Work in this area is already ongoing, and we’ve put a lot of thought into how to organise it in order to achieve a good result.
Research & design
Some of the functionality that is under review is a prominent part of GNOME 3, so it’s important that any changes we make are a genuine improvement. When it’s a core part of the desktop, even a small regression can be a significant issue for some users.
User research is going to be a key part of this process, both in order to ensure that we have a good understanding of user needs and requirements, as well as to test the outcome of design and development work, and modify it as necessary.
We have already done some initial, limited research, to find out how many windows, apps and workspaces are typically in use. The next research phase is currently being planned, and will involve a series of show and tell exercises, combined with semi-structured interviews, to get a better sense of how people use their desktops, and how the design can be improved for them.
Looking further ahead, we’ll conduct testing on any changes that we make, in order to evaluate how successful they are and ensure that users experience them as a genuine improvement over what came before.
At each stage, we hope to share the results of our findings.
Testing and iteration
Wherever possible, we are planning on landing UI changes incrementally, with an emphasis on maintaining a releasable product. This will allow us to pace our work and do testing and refinement throughout the development cycle, rather than just at the end.
When it isn’t possible to compartmentalise UI changes, we are planning on making development builds available early, in order to carry out testing and iteration over a longer period. This is likely to be the case for the bulk of any changes, due to the interconnected nature of the overview.
Watch this space
Design changes to GNOME Shell can generate both speculation and uncertainty. We’d like to mitigate this as much as we can, so that people understand what changes are coming, why they are being made, and why they can be confident that they are a real improvement.
Currently, there are a variety of experimental designs. However, we haven’t settled on a single proposal and don’t want to create false expectations by presenting them at this early stage. Hopefully once we have done more rounds of research we will be in a position to do this, and give a better idea of what UI changes might be coming further down the line. Until then, we ask you to be patient!
We also hope that the research, testing and feedback opportunities that we are planning will provide reassurance that any changes we eventually make will be positive. We are committed to make changes based on these data gathering exercises, if it turns out that the designs don’t perform as well as we’d hoped.
We will endeavour to provide progress reports on this work as it progreses, so watch this space for news!
During March, GNOME Shell and Mutter saw their 3.36.0 and 3.36.1 releases, and the beginning of the 3.38 development cycle. We’ve focused most of the development efforts on fixing bugs before starting the new development cycle.
From the development perspective, the 3.36.0 release was fantastic, and the number of regressions relative to the massive amount of changes that happened during the last cycle was remarkably small.
GNOME Shell
GNOME Shell saw continued improvements in its new Extensions app. New APIs were added to the Shell, which allows moving the Extensions app away into its own codebase. It also allows Shell to expose fewer interfaces through D-Bus. The Extensions app is now available on Flathub.
A number of other small bugs and crashes were fixed for 3.36.1. Notably, the blur effect now works properly with fractional scaling.
Initial 3.38 work includes an improved Bluetooth state reporting, and the usage of JavaScript promises to simplify various asynchronous operations.
Mutter
Following the 3.36.0 release, Mutter received various fixes to window streaming support. In contrast to streaming entire monitors, which was working properly, window streaming had a few quirks and misbehaviors. For 3.36.1, we’ve tracked down many issues around it and fixed them. Streaming windows is also done using DMA buffer sharing mechanisms.
On Wayland, sometimes new windows would use the wrong position to animate, leading to the zoom in animation look broken. This issue was fixed as well. Pasting images from Firefox does not freeze apps, specially Xwayland apps, anymore. We also fixed a series of bugs where Xwayland windows would show a black border when resizing.
Mutter now properly handles hardware cursors when hotplugging GPUs, and cursor hotspots now work correctly again on virtual machines. Sometimes cursors would rotate wrongly when on already rotated displays, and this was also fixed.
On the X11 front, Mutter now respects manually configured RandR panning on X11, and a bug preventing the correct monitor scale from being applied on X11 was also fixed.
Mutter now also finally respects the “middle mouse button” emulation setting exposed via GSettings.
As we enter the different feature freezes that come before the 3.36, development starts to wind down, and focus shifts to testing and fixing bugs. Nonetheless, this was an exciting month for Mutter & GNOME Shell! The changes that landed, just in time for the 3.36 release, range from code cleanups, new features, quality-of-life improvements, and preparations for future important features.
GNOME Shell
GNOME Shell’s CSS engine now supports auto. The icon grid spring animation was optimized to reduce the number of relayouts, which reduces CPU usage. Another batch of cleanups, refactorings, and fixes to GNOME Shell’s SCSS files landed.
Thanks to the ongoing efforts of extending and improving Sysprof, various bottlenecks and misbehaviors were identified in GNOME Shell and Mutter, and subsequently fixed. In particular, GNOME Shell now avoids doing I/O operations in the main thread when saving notification data on disk, and the local timezone is now cached, which avoids reading the contents of /etc/localtime more than necessary. As a consequence, GNOME Shell should behave better under heavy I/O loads on the host system.
The blur effect that is used by the new lock screen received further optimizations, and should be quick enough to not have any noticeable performance impact.
Animations are now disabled on various circumstances, such as when using a software renderer, when sharing screen with VNC streams, and when asked to by remote desktop sessions.
Extensions App
GNOME Shell now ships a new app to manage extensions:
The new Extensions application
This is now the primary way for users to manage extensions. It supports globally disabling extensions, uninstalling and updating user extensions, and toggling individual extensions on or off.
Mutter
Improved Screencasting Support
For a few years, Mutter implements a D-Bus API that allows screencasting. The screencast engine is implemented on top of PipeWire, and exports a PipeWire node that can be read and consumed by other applications.
Until now, this API would download the contents of the monitor framebuffers from the GPU to the system memory, and pass it to PipeWire. However, this was highly inneficient.
PipeWire 0.3 has support to another kind of buffer: DMA buffers. This allows Mutter not to download any framebuffer contents, and instead simply pass a file descriptor (an integer) to the clients. This results in far less CPU usage when screencasting.
A deep dive into how this was implemented will be published in the future.
Stage View Changes
Mutter is based on Clutter, which was originally developed as an application toolkit. Evidently, applications have very different constraints compared to compositors. One such constraint is about how refresh rates should be handled: applications only need to target a single refresh rate, whereas compositors need to draw on different monitors that may be running with different refresh rates.
These differences translate directly on how the code is architectured. Clutter has a single frame clock driving animations, and that does not play well with the problem domain that compositors live in.
During February, a major change on how Mutter handles drawing monitors landed. This is the first step towards achieving the goal of one frame clock per monitor.
Other Improvements
A stream of fixes to culling out rendering is being landed, and it potentially cuts down unnecessary rendering on various scenarios. For example, Mutter detects more cases when windows are completely obscured by other windows, and doesn’t ask them to render themselves.
A nice improvement to how the wallpaper is rendered allows Mutter to save resources when a scale is applied to any monitor.
Various fixes to Wayland subsurfaces support landed, and some small memory leaks were plugged. Lastly, Mutter now avoids flickering when X11 windows either ask to be unredirected or not.
The upcoming GNOME 3.36 release includes a major update to the system login and unlock experience. The new design has been anticipated for a long time, and we’re excited that it has finally arrived!
Some history
The lock screen in GNOME 3.6
GNOME’s existing login and unlock design has been largely unaltered since it was first introduced in GNOME 3.6, back in September 2012. That’s seven and a half years ago! It’s therefore no surprise that we’ve wanted to update the design for some time.
The initial round of design work for the new lock screen took place in 2017, at the GNOME UX hackfest in London. There, the GNOME design team, along with GNOME Shell developers, reviewed the goals and requirements, as well as the issues with the existing design, including the main areas of feedback that we’ve had.
Lock screen planning at the 2017 UX hackfest
The design concept that we came up with during that event has undergone significant refinement since 2017, but the basic principles and goals remain the same. These can be summarised in three words: smooth, beautiful and personal.
Reducing friction
Unlocking the system is something that people do over and over, so it’s important that it doesn’t feel frustrating, and it needs to get you from A to B as quickly as possible. One of the goals of the design update was therefore to reduce the amount of friction involved in unlocking (or logging in for that matter).
From this perspective, the most obvious change in the lock screen is that the grey login screen is now gone. Rather than removing the “shield” to show the password field, the password field is shown right on the lock screen itself. This doesn’t necessarily change how many keystrokes or clicks are required to unlock, but it does reduce how many distinct visual steps are involved, and makes the whole experience feel faster and more direct.
We’ve also incorporated other changes which are intended to make it quicker and easier to unlock. There are more methods to show the password field than before: you can click with a mouse or touchpad button. You can also scroll with a mouse, or swipe with a touchpad or touchscreen. Small changes like this add an extra degree of convenience in many situations.
Other changes in this area are planned for the future. One of the biggest is to jump straight to the last-logged in user after boot, rather than requiring a user account to be selected every time. In most situations, this will take an extra step out of logging in: all you’ll have to do is boot, then enter your password, and off you’ll go.
More happy times
If getting to your work more easily is one of the big themes for the updated design, then the other is having a more joyful experience along the way. Unlocking is a chore, but it shouldn’t have to feel like one. It can and should be pleasurable.
With the new design, we wanted unlocking to be beautiful, and we’ve put a lot of work into visuals. The most obvious change is the use of the blurred background, which we love. Not only does it look great, but it also serves as an effective background for legible, light-weight typography, so it’s possible to have elegant text, without the need for heavy type or drop-shadows.
There’s also been lots attention to detail throughout design and development process, which we hope will make the login and unlock experience feel delightful to use. This includes things like the updated layout of the authentication elements, the subtle use of animation to communicate incorrect authentication attempts, and the transition from the clock to the password field.
Making it personal
A final, and perhaps more subtle, aspect of the updated design is the effort to make it feel personal. We want to connect the login and unlock with the user’s desktop experience, so the user gets a hint of their session even when they are logged out or the system is locked. The main way we’re doing this is the use of the blurred wallpaper, which is intended to communicate what lies on the other side of login/unlock.
Mockup for the redesigned user selection screen
The use of strong personal elements, like the user avatar and blurred background image, is also something that we intend to build on in future versions, by bringing them into the user selection screen itself. The goal here is to bring user selection alive: to make it more expressive and indicative of the person to which each account belongs.
Getting to this point
On the development side, the login/unlock redesign has primarily been carried forward by Georges Stavracas, with significant assistance from Umang Jain and Florian Müllner. It has been a significant amount of work and we owe them a lot of gratitude for the time and effort they have invested in this project.
Next steps
The login/unlock redesign was merged slightly late in the GNOME development cycle. To compensate for this, it is undergoing an intensive testing and bug fixing effort. Help is needed with this, and there is a test plan available for those who would like to try the new design and report any issues they find.
As mentioned above, looking beyond GNOME 3.36, there are a number of aspects of the design which are yet to be implemented and which we hope to have for the next release, version 3.38, so there is hopefully plenty more to look forward to in this area.
One area of focus during this cycle was unifying the layout, content structure, and feel of dialogs in GNOME Shell. Many dialogs were redesigned, polished, and updated as a result of this effort:
Improved Gesture Support
The gestures to switch workspaces, and navigate through the application grid pages, were dramatically improved, and now follow the touchpad movement.
Password Peek
The password entries in GNOME Shell now support peeking passwords! Take a look:
Folders as Dialogs
A major change on how GNOME Shell handles application folders has landed this month. Instead of being displayed as popups within the application grid, folders are now displayed above it in the form of a dialog.
Miscellaneous
A new blur effect was added to GNOME Shell. It will be used by future work, such as a redesigned lock screen, among others.
Due to the VP9 encoder being too CPU-intensive, the built-in screen recorder changed back to the VP8 encoder.
As part of a code refactoring effort, the bits of StBoxLayout that deal with hiding overflowing content were moved to a new class, StViewport. This simplifies in various ways the implementation of overflowed content in GNOME Shell.
When managing the icons in the application grid, such as creating, renaming, or deleting folders, or moving app icons in and out of these folders, the icons will animate to their positions smoothly now.
GNOME Shell’s performance profiling framework was fixed and updated to also work on Wayland.
The SCSS code that is used to generate the default GNOME Shell theme was completely revamped and reorganized, making it easier to find specific theme classes. It also received a small visual update.
At last, GNOME Shell now supports NVidia GPU offloading by showing the “Launch on Discrete GPU” entry on systems with a NVidia discrete GPU available.
Mutter
More Cogl Cleanups
Cogl is the GPU rendering abstraction used by Clutter to render its contents on screen. Clutter itself is the base of Mutter and GNOME Shell, which add window management and compositing on top of Clutter.
During the past few months, there were continued efforts to cleanup and modernize Cogl by removing dead and unused code, using more modern OpenGL features, etc.
This work continued during December and January too, with more dead code saying goodbye, and more modern code being welcomed to the codebase.
Implicit Cogl API Removal
Due to historical reasons, Cogl has two major APIs: an explicit one, with properly defined types, and functions that receive the objects they should act on; and an implicit one, that is closer to the OpenGL model, and acts on an implicit state machine with the objects to be worked on.
Many parts of the Clutter, Mutter, and GNOME Shell codebases were using the implicit Cogl API. Those usages were updated, and allowed us to remove most of the implicit API from Cogl.
ClutterOffscreenEffect Improvements
ClutterOffscreenEffect is an effect applied on UI elements that allow them to be rendered in offscreen framebuffers. This allows interesting, shader-based effects to be applied to these elements, such as contrast and brightness, blur, among others.
A few bugs were found in this effect, and were ironed out. It also received some optimizations that allow the effect to release GPU memory more often.
Introduction of ClutterSeat
Clutter’s representation of input devices was designed in the X11/XI2 times, and the API was modeled after it. This change brings the internal input model closer to Wayland.
While it may sound an unimpressive change, it’s the cornerstone for an unification of the multiple input grabbing mechanisms in use by GNOME Shell.
Between other internal refactors that are now possible, it will also facilitate the introduction of an input thread in the native backend, so the pointer cursor is no longer frozen on stalls.
This will be covered in detail in a separate article.
Improve layout phase of no-layout actors
The Clutter toolkit follows the traditional rendering pipeline, composed of the layout, paint, and pick phases. The layout phase is where the elements (in Clutter terminology, “actors”) get to know their position and size; the paint phase is where these actors paint their contents on the framebuffer; and the pick phase is where Clutter figures out which actor is hovered by the pointer.
When we know beforehand that an actor is not going to move around the screen, we can tell Clutter about it, and the layout phase can be optimized to avoid some calculations. This mode is called “no-layout”.
During January, Clutter received another set of optimizations when dealing with no-layout actors. This can reduce CPU use on some specific scenarios, such as when dragging windows.
Miscellaneous
The ClutterContent interface now is better integrated in the size negotiation phase of ClutterActors, by making actors that follow their content’s sizes report their content’s geometry.
Many improvements to how Mutter tracks Wayland windows landed. Specifically, a crasher related to wl_subsurface was ironed out, a unit test framework was introduced, and a harmless runtime warning was fixed.
When dealing with multi-GPU setups, Mutter needs to share framebuffers across GPUs. There are currently three ways to do that:
The secondary GPU does an accelerated copy of the primary GPU framebuffer;
The primary GPU does an accelerated copy of its own contents to the secondary GPU framebuffer;
A slow, non-accelerated framebuffer is created on the secondary GPU, mapped to CPU memory, and the primary GPU contents are copied to this memory;
Mutter now features an additional way to share framebuffers: the primary GPU exports a framebuffer that is imported and read directly by the secondary GPU. This method is useful with virtual secondary GPUs, such as DisplayLink docks.
GNOME Shell saw many improvements during November. The commit log was dominated by cleanups, but a few improvements and polishments also found their way into the code.
The libcroco dependency was dropped by importing the source files into St. This is an important step in getting rid of libcroco, which is a dated CSS parsing library.
App Grid Improvements
The icon grid saw an important fix to dragging application icons. The icons were not properly being destroyed, and thus were piling up after dragging and dropping them over time. This fix was further improved to work on more situations. This set of fixes was backported to the 3.34 release.
A nice visual improvement landed on the page indicator of the icon grid.
For Mutter, November highlights were the introduction of regional clipping in Cogl, and big code cleanups.
Regional Clipping
When applications and GNOME Shell draw themselves, they communicate which parts of their contents changed. That information allows Mutter to submit only the changed contents to the monitor, which is an important optimization.
Example of GNOME Clocks being partially redrawn. The changed parts are painted in red.
Until GNOME 3.34, Mutter would calculate the bounding rectangle between all the regions that changed:
Mutter would submit the bounding box of all updated regions (in blue). In many situations, such as the above, that would include more than necessary.
This month, Mutter received the ability to update multiple regions independently, without using the bounding rectangle. In the example, Mutter now updates only what has actually changed:
The regions that Mutter submits (in blue) now matches the regions that really changed in the first picture (in red)
This yielded a significant improvement too! Under some circumstances, this change alone can reduce the time to submit frames by up to 44%.
Shadow Buffer
In some situations, in the native backend we now use a shadow buffer to render the stage off-screen before copying the content over to the actual buffer handed over to the display panel. While this may sound counter productive, it significantly increases performance from unusable to fairly pleasant on those systems that need it.
Other Highlights
We now prevent full window redraws when using dma-buf or EGLImage buffers on Wayland (mutter!948). This fixes partial updates of windows on Wayland, which can reduce the amount of data transferred between GPUs, CPUs, and the monitor. Together with the regional clipping explained above, this should significantly help saving battery.
Many, many Clutter and Cogl cleanups (mutter!921, mutter!819, mutter!933, mutter!932) landed too. These merge requests remove deprecated functions and features that, as time passes, are an increasingly burden for maintenance, and in some cases also prevent improvements and optimizations . About 28000 lines of legacy code has been cleaned out from Mutters own Cogl and Clutter versions so far, since we entered the 3.36 development phase. Extension authors, please make sure your extensions don’t use any of the removed code.
One legacy feature that dates back to when Clutter was a separate library used to write client applications was removed (mutter!911) from Mutter’s internal copy of Clutter. Not clearing the stage doesn’t make sense on a compositor.
Xwayland games that run fullscreen and change resolution should behave better now (mutter!739).
The last GNOME release, named “Thessaloniki”, was busy for GNOME Shell and Mutter. Many changes, ranging from code cleanups to architectural changes to performance improvements to new features landed.
Let’s take a look at the major highlights for the GNOME 3.34 release.
GNOME Shell
JavaScript Updates
GNOME Shell uses GJS, the GNOME JavaScript engine, to run. With the latest updates to GJS, such as the JS60 migration, GNOME Shell saw important updates making the codebase use modern tools and code practices.
Implicit Animations
One of the most important improvements that were part of GNOME Shell 3.34 was the transition to Clutter implicit animations. So far, GNOME Shell has been using the Tweener framework, which is written completely in JavaScript and, as a consequence, required extra work to communicate the animations to Clutter.
Using the implicit animations framework provided by Clutter allows GNOME Shell to skip a lot of JavaScript → C jumps, and thus, take less resources to run animations.
CI Integration
Testing GNOME Shell with CI is slightly tricky due to it and Mutter always being in lockstep. It is common that GNOME Shell from the git master branch needs Mutter from git master as well. Finding a good way to handle that prevented CI from landing earlier, but during this cycle we crafted custom container images and wrote the necessary scripts to allow for that.
Now, when running CI, GNOME Shell is tested with Mutter in tandem. Furthermore, on Mutter side, we’ve added a CI step to build and test GNOME Shell as well, catching many errors that would otherwise take some time to be found.
New Extensions Tool
GNOME Shell 3.34 also ships a new gnome-extensions tool, making it easier to create and manage GNOME Shell extensions. Due to the automation this new tool provides, such as extension templates, bash completion, packing, and testing extensions, it should be a significant improvement for extension authors.
Accessibility
With the ongoing transition to Wayland, GNOME Shell is taking ownership of many features that required random applications have full access to windows and their contents. One area where this bad practice is recurrent is the accessibility stack.
With GNOME Shell 3.34, a big coordinated effort was put in filling the missing accessibility bits when running a Wayland session, such as Locate Pointer, Click Assist, among others.
Click Assist menu. Courtesy of Olivier Fourdan.
Click Assist pie animation. Courtesy of Olivier Fourdan.
Folder Management
Last but not least, in GNOME Shell 3.34, it is possible to create, rename, and delete folders using Drag n’ Drop actions.
GNOME Shell 3.34 in Numbers
GNOME Shell 3.34 had 309 files changed, with 50,022 lines added and 26,924 removed. 59 merge requests were merged, and 1 was closed. 71 issues were closed by commits.
Sysprof is GNOME’s profiling framework, built on top of the `perf` API. With the GNOME 3.34 release, Mutter now integrates with Sysprof and is able to profile a lot of its internals.
Profiling Mutter is an important requirement to detect potential bottlenecks in the algorithms and the codebase, and have a proper understanding of how, where, and why things are the way they are.
GPU Hotplug
Mutter is now able to detect and use GPUs, usually external, that are plugged while the GNOME session is running.
KMS Transactions
In order for Mutter to support more complex hardware capabilities it needs to know how to utilize the atomic KMS kernel API. A first step in this direction was taken this cycle by implementing an internal transactional KMS API. Right now this API is backed by a non-atomic implementation, in order to make sure that we don’t break anywhere we currently fully function on, but in the future having this API will enable us to have mutter utilize more advanced hardware capabilities such as overlay planes and framebuffer modifiers. It is also a big step closer to being able to isolate KMS interaction into a dedicated thread eventually enabling low latency input device feedback once we move input processing into its own dedicated thread as well.
Improved Frame Scheduler
The algorithm that Mutter uses to schedule when to update the contents of the
monitor was fine-tuned and is now able to deliver new frames more smoothly. The new algorithm gives applications more time to draw their contents and notify Mutter about it. As a consequence, Mutter is much more likely to update frames in time for the monitor to display.
NVIDIA on Xorg frame throttling changes
Further in the past, Mutter used glxSwapBuffers() to throttle frame drawing when using the proprietary NVIDIA driver. This caused issues, as it’d block the main thread for long period of times if there was constantly new frames being scheduled. To mitigate this, Mutter developers introduced functionality imitating the GLX_INTEL_swap_event extension used by the Intel driver to make it possible for swapping buffers asynchronously, by receiving a “completion event” (swap event) later on.
This was done using NVIDIA specific GLX extensions combined with running a separate thread, where the “swap event” was generated, in contrast to when using GLX_INTEL_swap_event, where the Intel driver itself did this.
In practice, this caused issues, as it the relevant GLX extension implementation in the NVIDIA driver tended to be CPU heavy. However, due to the changes this cycle to our frame scheduling algorithm, the initial problem is now mitigated by scheduling frames in a way that we will we avoid blocking on glxSwapBuffers() for as long, meaning we could remove the CPU heavy “swap event” imitation code. This should mean less CPU usage when using the NVIDIA driver and displaying an application that continuously renders frames.
Graphene
Graphene is a library implementing complicated math functionality related to vertices, matrices, rectangles and other types usually related to 3D programming. In order to decrease maintenance burden on Mutter, which had its own set of complicated math functionality doing practically the same thing, we went and replaced most of that code, and switched to using Graphene directly. Not only did we decrease the amount of code we have to maintain, we also got all the fancy new improvements done to Graphene that we would otherwise be without!
For GNOME 3.34, all but matrices were converted to the corresponding Graphene types. That’s because Cogl and Graphene matrices have different semantics and, thus, will require more scrutiny.
XWayland on Demand
Mutter and GNOME Shell 3.34 earned the ability to run as a pure Wayland compositor, while still providing seamless compatibility with X11 by starting XWayland when required by applications. This is the culmination of much prior effort at refactoring internals and ironing out indirect X11 dependencies throughout Mutter and GNOME Shell specifically, and the GNOME session generally.
Being able to start Xwayland and other related X11 services on demand for legacy applications came out organically, and demonstrates we are no longer tied by our legacy support.
This feature is disabled by default, and can be enabled by adding ‘autostart-xwayland’ to `org.gnome.mutter experimental-features`.
Real-Time Scheduler
When running as a Wayland compositor and display server, it is important that Mutter keeps its reponsiveness all the time. However, Mutter runs as a regular user application, and is subject to starvation in various scenarios, such as intensive I/O.
Mutter 3.34 now requests real-time priority, which gives it more priority than
regular applications. This avoids stalls and stutterings when running GNOME Shell as a Wayland compositor.
This feature is disabled by default, and can be enabled by adding ‘rt-scheduler’
to `org.gnome.mutter experimental-features`.
MetaShapedTexture as a ClutterContent
MetaShapedTexture represents the visual contents of an application’s surface. It takes care of making sure the content is cropped and scaled properly, and tries to optimize drawing depending on what is opaque, and what is not.
ClutterContent, on the other hand, is Clutter’s way of defining deferred rendering instructions, also known as render nodes. This enables more efficient batching of draw calls, compared to the implicit batching implemented by the Cogl journal. This cycle we took the MetaShapedTexture, which was a ClutterActor subclass manually issuing Cogl drawing instructions, and turned into a set of fully deferred rendering instructions as defined by ClutterContent.
This not only improved batching of draw calls but greatly decreased the size of the Clutter actor tree, giving us a significant performance boost.
Here’s a quick comparison:
Turning MetaShapedTexture into a ClutterContent implementation (green bars) reduces frame times when many windows are open, compared to the previous implementation (yellow bars). Courtesy of Jonas Dreßler.
Mutter 3.34 in Numbers
Mutter 3.34 had 766 files changed, with 34,249 lines added and 37,268 removed. 74 issues were closed by commits.
| 122 | Jonas Ådahl |
| 109 | Carlos Garnacho |
| 82 | Marco Trevisan (Treviño) |
| 52 | Olivier Fourdan |
| 48 | Georges Basile Stavracas Neto |
| 40 | Florian Müllner |
| 30 | Adam Jackson |
| 27 | Daniel van Vugt |
| 24 | Hans de Goede |
| 24 | Robert Mader |
| 19 | Jonas Dreßler |
| 16 | Niels De Graef |
| 16 | Pekka Paalanen |
_showing_search_function_in_overview_mode.png){kind=link}