If you’ve been following this blog, you’ll know that a team has been working on updated designs for the Activities Overview. (Previous posts on this topic covered our initial motivations and design goals, as well as the results from some early exploratory research that we conducted.) This initiative has been the subject of significant activity over recent months, and we’re now at a point where we can share more details about what we’ve been doing.
Where we’re at
Following months of design exploration and 6 separate user research exercises, which included a study by a user research firm, the GNOME Shell team has an updated design for the Activities Overview, which it hopes to land in time for GNOME 40 (scheduled for release in March 2021).
All the user research has given us confidence in the design. The core aspects of the latest design all performed well, and we’ve observed a general preference for the new design in our latest internal research exercise. That said, we’ll be continuing to review the design as feedback comes in.
During the design and testing process, the designs have been in state of steady evolution. Now that they are more stable, we’re excited to share them and invite contributors to help us get the new design over the finish line!
In the rest of this post, I’ll briefly describe the design in its current form, before going on to outline the roadmap for what’s going to happen between now and GNOME 40. In subsequent posts, members of the team will provide more detail on the design itself and the research that has been conducted.
We’ve discussed the goals for this initiative a couple of times in the past (see here and here), but let’s refresh our memories. The things we wanted to achieve with this effort include:
Better overview spatial organisation
Helpful boot and empty states
Improved touchpad navigation
More engaging app browsing and launching
We feel that the current iteration of the design delivers on these goals, while retaining the core features of the existing GNOME Shell design that users enjoy today, including a distraction-free desktop, super key to enter the overview, and type to search.
A reimagined Activities Overview
The new design reimagines the Activities Overview using a strong spatial metaphor. Workspaces are arranged horizontally, and have a physical quality to them. (In tests, participants found this easier to understand, as well as being more engaging.)
Workspaces appear in a continuous sequence, from left to right. This sequence can be panned and scrolled. Windows have app icons on them to help with identification, with the window title being shown on hover.
A good deal of work has already gone into the app grid, so that the new design is fairly similar to what is already in the last GNOME release (3.38).
The key feature of the new grid is its customizability. Launchers can be re-arranged at will, both within multiple pages as well as using folders. This interaction was immediately intuitive to testers.
The main difference between the new design and current GNOME is the orientation of the app grid pages, which are now horizontal rather than vertical, as part of the general spatial reorganisation of the overview.
By using a strong spatial model, the design aims to support intuitive navigation and orientation. This is the primary motivation for changing the orientation of workspaces from vertical to horizontal: with the workspaces laid out horizontally, the vertical axis can be used to control movement in and out of the overview and app grid.
This spatial arrangement allows navigation to operate as if in two dimensional space: left and right moves between workspaces, up and down enters the overview and shows the app grid. This is particularly powerful when navigating with touchpad gestures (four finger swipes, up/down/left/right) or keyboard shortcuts (Super+Alt+↑/↓/←/→).
Work is ongoing to implement the design and work through outstanding design issues. Implementation is planned to take place as a series a separate branches which will land in gnome-shell master. Thankfully, much work has been done to make the overview code more flexible, which will make this process easier.
Ensuring excellent quality is a high priority for us. Therefore, we plan on making development builds of the changes available to test as early on in the process as possible. The more people we can have running the code and providing feedback, the better. If you’d be interested in testing development builds, look out for follow-up posts.
It is hoped that these changes will be the start of a series of improvements to the design of the shell that will arrive in subsequent releases. Certainly, the research work that has been done so far has given is a vast amount of insight that can’t all be addressed in a single release.
Being a compositor and a compositing window manager, the most important aspect Mutter and GNOME Shell is to paint pixels to your monitors with relevant content. A large part of this content is provided by applications themselves, but many elements still need to be rendered on top of them.
Over the past few years, Mutter’s codebase has slowly but steadily been refactored, cleaned up, reorganized, and modernized. This includes the internal copies of Clutter and Cogl. With the beginning of the GNOME 40 development cycle, it all converged in a specially large and exciting set of changes which we’ll be talking about in this article.
At its core, Mutter inherits the rendering routines of Clutter and Cogl. Cogl itself is a layer above OpenGL / GLX / EGL / GLES, which is what it ultimately boils down to.
An interesting aspect of Clutter is that, despite seeming like a 2D toolkit, it actually renders actors in a 3D space. This is what allows effects like moving, scaling, skewing, and rotating actors.
Clutter uses a traditional perspective projection when rendering. In practice, this mimics the real world where whatever is closer appears bigger, and what’s farther appears smaller. GNOME Shell (the “stage”) itself is rendered in a well-defined position inside this 3D space: the Z-2D plane.
The camera in Clutter is an immutable, implicit camera positioned at the center of the 3D world, and many of the traditional techniques used in rendering engines also apply here, such as the Z-near and Z-far planes (in the illustration above, the planes delimiting the blue region of the view).
An interesting technique that is familiar to game developers is ray casting. In our case, ray casting is used to detect which element on screen is beneath the cursor, a process known as picking.
Since times immemorial, Clutter used painting for picking. Actors would paint themselves on a frame buffer, each actor with a different color, and Clutter would read a single pixel of the resulting picture. Cogl had several optimizations to try and not send these operations to the GPU through a journaling mechanism, but due to various constraints of how Clutter worked, these optimized code paths weren’t hit very often.
With the 3.34 release, Clutter moved away from this model of picking, and started using geometric picking. Not touching the GPU while picking, and (most importantly) not downloading GPU contents, was a considerable improvement. However, it still involved quite a bit of vertex projection, a relatively heavy operation that should be avoided when possible.
The well established technique of raycast throws a virtual “ray of light” into the scene, and does a hit test against a list of rectangles to figure out which ones are touched by the ray.
One fundamental aspect of ray-based picking is that it can project vertices only when necessary to perform the hit test. In best-case scenarios, it only projects a single group of vertices. In worst-case scenarios, it projects as much as it used to do. Furthermore, by using the graphene_ray_t helper, we also benefit from vectorized operations. These facts led to a measurable reduction in how much time it takes for each pick operation to be executed, and the relative percentage of the frame times it takes.
These refactorings of the pick code enabled another improvement: pick culling. Much like culling out actors when painting, we can now cull out actors when picking. Initial profiling sessions show that, together, these improvements massively reduce pick times.
While painting what you see on your screen is a fundamental aspect of GNOME Shell, figuring out what not to paint is almost as important!
The process of “culling out” elements from the rendering pipeline is an important optimization. When you hover a button inside an application, for example, it is common that the application will only redraw that specific button, and will tell Mutter which region of it needs to be updated on screen. This allows Mutter to only redraw the region of the screen that should be updated on the next frame.
We’ve written about this feature (regional clipping) before, in a previous article. However, this cycle, we introduced an important improvement on top of it: clip frusta.
Despite the intimidating name, the concept is familiar: much like a clipping region is a set of 2D rectangles representing which parts of the screen contents must be updated, the clip frusta is a set of 3D volumes (frustum) representing which slices of the 3D space must be updated.
These volumes are built by projecting the 2D clip regions in the 2D plane, and extending them all the way to the camera, and also all the way against the camera. They are also delimited by the Z-near and Z-far planes.
By representing clips in 3D space, we can avoid projecting 3D actors in 2D planes over and over, which is yet another significant optimization of the rendering process.
Thanks to Graphene, we have access to graphene_frustum_t and various routines to operate on it. This allowed us to simultaneously optimize rendering, and remove code!
You may have noticed that we’ve mentioned Graphene multiple time throughout this article. Thanks to this fancy library, a set of data structures and algorithms is readily available for us to use.
In addition to frustums and rays, Graphene offers methods and data types for 2-, 3-, and 4-dimensional vectors, boxes, rectangles, triangles, and various onlyothers. Despite using various types from Graphene, only recently Mutter moved away from its own internal implementation of matrices (CoglMatrix) to graphene_matrix_t, which allowed for another 4-digit code cleanup.
Now that we’ve dropped many core components of Mutter, Clutter, and Cogl, in favor of their Graphene counterparts, it is possible to think about other improvements, specially in the rendering pipeline itself.
There already is preliminary work improving the painting routines to use paint nodes more extensively, porting effects to this new API, input processing improvements, and others that directly or indirectly benefit from using Graphene. Stay tuned!
It’s been a while since we last blogged about the GNOME Shell design work that’s been happening. While we might not have blogged in a bit, there’s been a lot going on behind the scenes, particularly on the research side, and it’s about time that we told everyone about what we’ve been up to.
As a side note: a great team has developed around this initiative. The existing design team of Jakub, Tobias and myself has been joined by Maria Komarova from System76. Maria has a particularly strong research background and was immensely helpful in running interviews. The development side has also been fully engaged with the process, particularly through Georges and Florian.
Research and goals
In addition to thinking and discussing the design goals that we outlined previously (here and here), the design team has conducted two research exercises over the past six months. The main research exercise was a series of interviews with existing GNOME users. We also ran a small survey, to get some baseline numbers on user behaviour. I’ll mostly be talking about the interviews here.
The goals of the interviews was to a) get general information on how the shell gets used, b) evaluate how the existing shell design is performing, and c) identify ways in which the shell could be improved for our users. In addition to these fairly specific goals, we also wanted to develop a deeper understanding of our users: how they use their computers, their concerns, predispositions, interests and attitudes.
Interview-based research is a classic research approach for answering these kinds of questions, which are particularly appropriate to ask early on in a design process, when the emphasis is on exploration, insight and understanding. (Some readers might be more familiar with usability testing as the means through which data-driven design gets done, and that is a useful approach, but it typically happens later in the process.)
The survey element of our research was a smaller exercise than the interviews, so I won’t go into it in much detail. It was solely conducted within the confines of Red Hat’s Desktop Team, purely for convenience’s sake, and was able to provide some initial numbers for use by the design team.
The survey was an opportunity to ask users about their number of running apps, open windows, workspace usage, and a number of other behavioural factors. The key goal of the exercise was to identify typical workloads as well as the extremes, in order to inform our design requirements around the shell and the overview.
For example, through this exercise, we found out that the modal number of open windows was 8, and the highest number was 28. We also found that most people (63%) were using a single workspace, and that the largest number of workspaces in use was 10.
Clearly, such a limited exercise has to be treated with a high degree of caution, but some quick preliminary numbers were a useful frame of reference and are something that we hope to build upon in the future.
The rest of this post will be devoted to the interview exercise that we conducted. Here’s what we did…
We conducted seven interviews in total, each conducted remotely. Interviewees were recruited from through the team’s professional contacts. We avoided talking to anyone who is directly involved in desktop development, and we tried to recruit research participants with a range of roles and technical backgrounds. We had some developers, but we also had people who work in sales, marketing and support.
During each interview, we got general background information about the interviewee and their computing activities. Then we conducted a screen sharing exercise in which the interviewee showed us how their desktop was set up. Then we ran through some exercises to see how they performed basic tasks.
We also used the interviews to ask evaluative questions, to see what the interviewee thought about the current shell design, and how it compares to other desktops.
We took notes on each interview. We mapped out the responses we got. We identified themes and patterns.
Let’s start by talking about the user behaviours that we observed. A quick summary:
The number of apps in use was generally low, 3-5 being the common range, both in the interviews and the survey.
Search was prominent as a way of launching apps, but we also observed occasional pointer-driven app launching (curiously, this seemed common when launching some apps and not others).
Engagement with the app grid was very low.
About half the interviewees had customised the favourites in the dash.
Workspace usage was quite mixed: some used workspaces extensively, while others never touched them [1, 2].
These findings have direct relevance to the design areas that the team are investigating. For example, they imply that our designs need to cater both to users who use workspaces extensively, and those who don’t.
The results also posed some conundrums. For instance, is lack of engagement with the current app grid an indication that its design needs to be improved, or that the feature should be demoted or relegated, somehow?
Likes and dislikes
While much of the research findings related to observed behaviours, the interviews also allowed us to ask evaluative research questions. We wanted to know how the existing GNOME Shell design is performing, and how it can be improved.
The main thing that came across here is that overall, the existing design appeared to be working well for the users we spoke to. With one exception, all the interviewees thought that the design compared favourably with other desktops they’d used. (Of course, this insight needs to be taken with a touch of salt .)
Most of the interviewees identified things that they like about the current shell design. They seemed to really like the search feature, in particular its speed, but also that it’s just a single keystroke away. The overview also received regular praise (one interviewee described it as “fantastic”).
The other thing that repeatedly came up as a positive aspect of the existing design is its none-distracting nature. Interviewees paid the shell compliments by saying things like “it’s not cluttered” and “it gets out of my way”.
The main issues with the current design that came out of the research was the frequent lack of engagement with the dash and app grid. We also got a hint that onboarding might be a struggle for some, and we got a clear message that touchpad gestures need to be improved (confirming one of our initial hypotheses).
We are already planning additional rounds of research (which we will endeavour to blog about in a more timely fashion!) Some of these will be open to wider participation so, if you are interested, we would love to have you involved!
The design team is also exploring various concepts for the shell, which we hope to present in the not too distant future. Indeed, one of the reasons for all this research work is to help us be confident about the designs before we present them to the world! Watch this space.
 There was also behavioural variation within the group of workspace users: some had developed systems for themselves, with certain workspaces dedicated to particular activities, while others allowed their workspace usage to evolve organically as they worked.
 Workspace usage did not appear to be positively related to technical expertise. If anything, the opposite was true, with the less technically-expert making more use of workspaces rather than less.
 Some notes on: representativeness and generalisability. How can only seven users provide useful results? Won’t we have missed critical user types? Won’t the sample be unrepresentative?
The first thing to say here is that small-N studies (those with a small number of participants) are valuable and can deliver significant insights, and have important advantages over large-N surveys. In particular, they allow accurate observations to be made, without making assumptions about what’s going to be seen.
Second, small-N studies like this allow relationships to be identified, and these can be used to infer qualities of a wider population. For example, if you observe that technically expert users tend to use the keyboard more, and you know that you have a lot of technically expert users, you can make a guess that keyboard use is important for the population as a whole.
Third and finally, experience tells us that small research studies can and do pick up on general trends and behaviours with remarkable speed (such as with the rule of five in user testing).
That said, we certainly do need to acknowledge the non-representative nature of our interview sample. We did include some relatively non-technical users, but the sample as a whole was skewed towards the more technical/professional end of the spectrum, and was taken from organisations that are invested in the Linux/GNOME desktop to varying degrees.
This doesn’t mean that the results we got are invalid. We are confident that our research findings are accurate with regards to the people we spoke to, and that those people are indicative of a larger section of the GNOME user population. The only question is is whether there are other types of user who we didn’t include.
The past two months were hectic, and many changes and new features landed just in time for the 3.37.90 release. As the release freezes approached (user interface freeze, API freeze, etc), part of the queue of merge requests needed a final decision on whether or not they were ready for the upcoming GNOME 3.38 release.
These two months were also punctuated by not one, but two conference talks related to Mutter and GNOME Shell.
As this is the final development summary before the 3.38 release, we decided to change the format a bit and cover the biggest changes that happened throughout this cycle.
Split Frame Clock
Back when Clutter was an application toolkit, it only had to deal with a single frame clock. However, now being a compositor toolkit, having a single frame clock became a limiting aspect of Clutter, since each monitor is ticking at potentially different rates and times. In practice, that meant that on multi-monitor scenarios, the monitor with the slowest refresh rate would limit other monitors.
A large surgery on Mutter’s fork of Clutter implemented a new frame clock, and made each ClutterStageView create its own frame clock, and that allowed monitors not to interfere with each other.
Applications such as games, media players, etc, allow running in fullscreen mode. Ideally, when an application is fullscreen should be able to avoid needlessly compositing the desktop when the only visible thing is the client’s fullscreen window. This is usually called “compositor bypass”, or “fullscreen unredirect”.
During this cycle, Mutter on Wayland supports bypassing the compositor when possible. When bypassing the compositor, the window’s content is put directly on screen, without any unnecessary compositing. The results vary from hardware to hardware, but in principle this should reduce CPU and GPU usage and, consequently, improve performance.
Another bunch of screencast improvements made its way to Mutter. Noticeably, screencasting now works correctly even when an application is bypassing the compositor, in which case Mutter directly copies the contents of the application directly to the screencast stream. Have a look:
In addition to that, window screencasting received a round of bug fixes and improvements.
At last, GNOME Shell’s built-in screen recorder was split into a separate system service, which should make recording your screen using it smoother.
Customizable App Grid
A big change in GNOME Shell will be part of GNOME 3.38, and that’s the ability to customize the app grid. This feature was a long wish, and required a substantial number of under-the-hood improvements, and other changes to support it.
It currently supports creating folders by dragging application icons over each other, moving applications from and to folders, and repositioning applications inside the grid.
Folder dialogs also received small visual improvements to match these new changes.
Despite being a feature in itself, the customizable app grid is a necessary step to unlock future design changes that we’ve been researching and investigating for some time now. Stay tuned as we release more information about it!
Calendar Menu Updates
The calendar menu received a handful of visual improvements as well. Calendar events are now displayed below the actual calendar, and sections as more visually prominent.
There are more improvements for this menu queued up for the next release. Google Summer of Code student Mariana is continuing to work on grouping notifications, and we expect it to be ready for the release after next.
Various components of the desktop, including GNOME Shell, Settings, and others, now optionally integrate with the parental controls service that is part of GNOME 3.38.
This allows parents, guardians, supervisors, schools, among others, to limit which applications can be accessed by a particular user.
The shutdown menu was split, and “Restart” is now an entry, together “Shutdown”:
The layout algorithm of the app grid was rewritten, and should improve the icon arrangement on various scenarios. The number of rows and columns is now defined based on the aspect ratio of the monitor, and the available space, and the icons themselves grow and shrink accordingly.
The number of icons per page is fixed at 24 icons per page. That’s because changing the number of icons per page would potentially incur losing the customizations made to the app grid.
In addition to that, both Mutter and GNOME Shell received an influx of under-the-hood optimizations and improvements throughout the cycle, ranging from Cogl, the underlying OpenGL layer that Mutter uses internally, to Clutter and it’s layout machinery, to Mutter itself and GNOME Shell. Together, these changes bring polish and robustness, and add up for small but noticeable improvements to the experience of using GNOME on a daily basis.
GUADEC is the traditional, yearly GNOME conference. During the 2020 edition of this conference, Mutter & GNOME Shell contributors and maintainers had the chance to present the also traditional “State of the Shell”, where a recap of the past 2 cycles was made:
We appreciate the efforts of the GUADEC volunteers that made this year’s edition possible online.
Linux Plumbers Conference
In the 2020 edition of the Linux Plumbers Conference (LPC), we had the opportunity to be part of the debut of the Applications Ecossystem Micro-Conference, together with the Flatpak and Plasma Mobile projects:
We highly recommend everyone to watch and appreciate all three LPC talks in this video, for they contain lots of valuable and interesting information.
Congratulations for the LPC team for providing such a great online conference.
This release cycle was stuffed with new features, improvements, cleanups, code refactorings, and bug fixes, and we had a great time working on it. It is exciting to see such a large number of changes be part of a release, and we are proud of the advancements that we maintainers, contributors, and independent volunteers managed to achieve.
The results of our focus on GNOME Shell and Mutter have been really pleasing, but not just to us! Here are some folks who have great thoughts about it:
“The GNOME Shell team continues its steady path of technical and user experience improvements – not an easy thing to do in a codebase with diverse uses and a long history, such as GNOME Shell / Mutter. The team also sets an example for very successful cooperation between multiple companies and the community. Well done!”
— Matthias Clasen, GTK maintainer, Red Hat
“It’s incredible how polished and mature GNOME has become. I’ve used it to convert many new users away from Mac and Windows and I’m still looking forward to the future! Thanks GNOME team!”
— James (purpleidea)
As we celebrate the imminent GNOME 3.38 release, we also look forward the release after that, and contemplate the many challenges lie ahead.
Readers be advised, this is somewhat of a deep dive into the guts of Mutter. With that out in the open, lets start!
Not too long ago mutter saw a merge request land, that has one major aim: split up the frame clock so that when using the Wayland session, each monitor is driven by its own frame clock. In effect the goal here is that e.g. a 144 Hz monitor and a 60 Hz monitor being active in the same session will not have to wait for each other to update, and that the space they occupy on the screen will draw at their own pace. A window on the 144 Hz monitor will paint at 144 Hz, and mutter will composite to the monitor at 144 Hz, while a window on the 60 Hz monitor will paint at 60 Hz and Mutter will composite to the monitor at 60 Hz.
All of this is roughly achieved by the changes summarized below.
In the beginning of times, Clutter was an application toolkit. As such, it assumed (1) the existence of a window compositor, and (2) the compositor is a different process. Back then, Wayland was still in its early infancy, and those assumptions wouldn’t conflict with writing a X11 window manager. After all, a X11 window manager is pretty much just another client application.
Over time, however, Clutter started to grow Wayland integration in itself. Deeper and deeper surgeries were made to it to accomodate it being used by a Wayland compositor.
In 2016, the Cogl and Clutter codebases were merged with Mutter codebase, and they all live in the same repository now. However, to this day, relics from the time when Clutter was an application toolkit are still present in Mutter’s Clutter. One such relic is ClutterMasterClock .
ClutterMasterClock was the main frame clock that drove Clutter painting. As an application toolkit, only a single, global frame clock was necessary; but as a compositor toolkit, this design doesn’t fit the requirements for multi-monitor setups.
Over the last cycles, there has been some attempts to make it handle multiple monitors slightly better by juggling multiple monitors with their own refresh rates and clocks using various tricks, but the fundamental design was standing in the way for making substantial progress, so it has been completely decommissioned.
ClutterFrameClock is the new frame clock object that aims to drive a single “output”. Right now, it has a fixed refresh rate, and a single “frame listener” and “presenter” notifying about frames being presented. It is also possible to have multiple frame clocks running in parallel.
However, ClutterFrameClock alone isn’t enough to achieve independence of monitor redraws.
Mutter has a single stage that covers the union of all monitor rectangles. But how does it render different contents to each one of them?
That’s one of the main responsibilities of ClutterStageView.
ClutterStageView was the answer to the need of drawing the stage at different framebuffers. ClutterStageView corresponds roughly to one monitor. Each ClutterStageView holds the on-screen framebuffer that the monitor displays; if using shadow framebuffers, ClutterStageView also handles them; and finally, it also handles the monitor rotation.
Now, ClutterStageView also handles the monitor’s frame clock. By handling the frame clock, each view is also responsible of notifying about frames being presented, and handling the frame clock dispatching
The frame scheduling related logic (including flip counting, schedule time calculation, etc) was spread out in ClutterMasterClockDefault, ClutterStage, ClutterStageCogl, MetaRendererNative, MetaStageNative, and MetaStageX11, but has now now been concentrated to ClutterFrameClock and ClutterStageView alone.
Actors, Actors Everywhere
When animating interface elements, the core object that does that is ClutterTimeline and its subclass, ClutterTransition .
Timelines and transitions saw frames whenever the master clock ticked. With the master now clock gone, they need to find an appropriate frame clock to drive them. In most (and after this change effectively all) cases a timeline was used to directly drive an animation related to an actor. This indirect relationship is now made explicit, and the timeline uses the actor to find what stage view it is being displayed on, and with that information, picks an appropriate frame clock to attach to.
For transitions, used extensively by GNOME Shell to implement animations, this is handled by making a ClutterAnimatable provide the actor, and for stand-alone timelines, it’s a property set directly on the timeline before it’s started.
This means that when an actor moves across the stage and enters a different stage view, the timeline will be notified about this and will decide whether to migrate to a different frame clock.
What About X11?
In the X11 session, we composite the whole X11 screen at once, without any separation between monitors. This remains unchanged, with the difference being where scheduling takes place (as mentioned in an earlier point). The improvements described here are thus limited to using the Wayland session.
Be aware of API changes
To be specific, for extension developers, there are two things to keep in mind:
If you use a St.Adjustment. You must now pass an actor when constructing it. This actor will determine what frame clock will drive the adjustment.
Some signals saw their type signatures change, namely ClutterStage::presented, ClutterStage::after-paint.
This is a big achievement to Mutter, GNOME Shell, its users, and especially to the contributors that were part of this. The road to reach this point was long and tortuous, and required coordinated efforts of dozens of contributors over the course of at least 5 years. We’d like to take a moment to appreciate this milestone and congratulate each and every single contributor that was part of this. Thank you so much!
The volunteers and contributors working on Mutter and GNOME Shell have been busy in the past couple of months — so much so that we didn’t have bandwidth to write the May development report!
As a consequence, this development summary will have an above average number of changes to highlight.
Preparations for Customizable App Grid
As part of the preparations for a customizable application grid, a new layout manager was written and replaced the current icon grid code. This new layout manager is in many ways more suitable for current and future changes:
It follows the delegation pattern that is common to Clutter. As such, it is a layout manager, and not an UI element itself.
It allows more precise control over the how the grid is displayed.
The most visible impact is that it now selects a row x column configuration that is closest to the aspect ratio of the display:
There are still improvements to make, especially with ultra-wide displays, but the foundation work is already there, and it will be vastly easier to fine-tune the behavior of the app grid on different scenarios.
An unfortunate oversight was causing the Do Not Disturb setting to be reset on startup. This bug was fixed. A potential D-Bus race condition when creating MPRIS media players was corrected. App icons do not vertically stretch in the top bar anymore. These bugfixes were backported to GNOME 3.36.
The code that deals with workspaces in GNOME Shell is old, but a large number of cleanups to it has landed (!1119, !1251, !1294, !1297, !1298, !1307, !1310, !1313, !1320, !1333), and even more is under review. These cleanups were much needed in order to improve the overall quality and maintainability of the codebase.
The Extensions app saw some improvements too. The Logout button now workscorrectly.
When the host system changes timezones, GNOME Shell now properly updates the timezone offsets of the “World Clocks” section of the messages popover.
Last year, MetaShapedTexture was made into a ClutterContent implementation. This change was important for a multitude of reasons, and will play a special role in the future with upcoming cleanups. However, it also introduced an unforeseen regression: Clutter paints ClutterContents before running the main painting routines, and this broke the existing culling mechanism of Mutter. After some investigation, culling wasfixedagain.
Mutter also received a seriesofimprovementstoitstestsuit. These improvements range from fixing broken tests, make CI more reliable, add more tests, reorganize the entire test suit, among other changes.
This is the second of a series of posts on the design of GNOME Shell, as part of an ongoing initiative to improve the core shell experience. In the previous post of the series, Allan gave a general introduction to that initiative. This post is meant to provide some more context and touch on longer-term ideas we’re thinking about. It’s worth noting that many of these are still at an early stage, and subject to change.
It’s hard to believe, but we’re coming up on 10 years of GNOME Shell. Though GNOME 3.0 wasn’t released until April of 2011, design and development were in full swing in mid-2010, and had been for some time.
Of course, the shell has not been standing still since then. Many important areas have been refined and iterated on since those early days, including the notifications, the app grid, and the system status area.
However, other parts of the interface haven’t changed much since the early days, and we know that some aspects of these old designs could be improved. There are a number of core elements of the shell that we’ve wanted to refresh for some time now, and as the previous blog post discussed we’re hoping to move forward with some of it in the near future.
Given that background, this post is meant to provide an overview of where we see the strengths and weaknesses of the existing design, as well as areas that we’re interested in improving. This includes the areas highlighted in Allan’s post, but also expands the list to cover other initiatives that are either ongoing, or which we would like to see happen.
One of the core ideas behind the original design of the shell was getting out of your way, and reducing distractions as much as possible. While you’re using apps there are no colorful icons, applets, or other clutter on the screen: It’s just you and the thing you’re working on.
Long before big tech started talking about giving people back control over their attention, “digital wellbeing”, and “time well spent”, GNOME Shell was built around helping people deal with distractions and focus on what really matters. Needless to say, this is more relevant than ever, and something we want to continue to focus on.
If you’re curious, here’s some further reading on the thinking behind the initial 3.0 shell design:
The GNOME Shell overview is a single place to go for app launching and switching, window management, and system-wide search. It’s the center of the shell experience, the place to see what’s happening on the system at a glance, switch activities, or start new ones.
No other desktop OS has a core workflow this simple and straightforward, and many people really really love it. This is another core strength we want to build on.
Most operating systems have some kind of system-wide unified search nowadays, but the GNOME Shell one is especially great because it’s seamlessly integrated with the “single entry point” workflow. This means there’s no extra button to click, or separate keyboard shortcut to learn. Just hit Super and type.
Switching > Launching
Most of the time when leaving the current activity you don’t start a new one (e.g. by opening a new app), but switch to another running activity. GNOME Shell optimizes for this by putting multitasking front and center in the overview, ahead of launching new apps.
A key reason for the success of the “single entry point” workflow is how easy it is to open the overview across input devices (most of them anyway, but more on that later).
With a mouse you just flick your cursor up to the top left corner. This is very ergonomic because you don’t have to aim, you just move the mouse in that general direction.
If your hands are on the keyboard you can hit a single key, conveniently located in the bottom left of the keyboard, to enter the overview.
These methods quickly become second nature, so much so that people report feeling a visceral lack of control when using other systems. Windows and macOS have equivalents to many of GNOME Shell’s individual features, but there’s no way to do it all from one place in such an ergonomic way.
Areas for Improvement
While there’s a lot to like about the status quo, we’re aware that there are also a number of issues with it, both structurally and usability-wise, and there are a lot of things that work well today, but we think could be even better.
Launching & Initial State
One of the most visible shortcomings of the current shell layout is the bootup experience. After login you’re greeted by an empty desktop, and when you open the overview it’s also empty, since there are no open windows. Though this is not something you encounter very often, it’s definitely not a good first impression.
There are also issues with launching apps from the app grid more generally: It’s not very prominent or easy to get to, and it’s split into two separate views, each with its own problems. Apps don’t have fixed positions in either view, because the All view is alphabetical, so things move around when you install or remove apps, and the Frequent view constantly changes depending on usage.
In improving the app grid, we’d like to move towards something that allows people to develop spatial memory for where certain apps are, and organize them in a semantic way.
Our current multitasking model consists of two levels: workspaces, which are laid out next to each other in a vertical row, and windows, which are stacked behind one another. This two-level system can be quite powerful, but there’s no automatic onramp to the workflow. By default, if you launch apps they all just pile up on the same workspace.
It’s possible to use GNOME Shell with just a single workspace, but it can be awkward, e.g when there are too many windows for the overview to be an effective way of switching because the thumbnails are so small. All windows stacking up behind one another also makes it hard to do things that involve multiple windows at the same time, such as keeping two windows side by side.
People often mention that once they were shown the workspace feature (e.g. by a friend or coworker), it “clicked” for them and they started using and loving it. Of course, this kind of discovery doesn’t scale, and there’s clearly a gap in the learning curve.
This issue isn’t new for workspaces (across all implementations, not just ours): You need to know about the feature, actively decide to use it, and put in some manual work. In our case this is made worse by the fact that the system isn’t able to “remember” open windows and workspaces across reboots.
Ideally we’d like to make multitasking and window management feel more organic than the current two-level system, and reduce the amount of work people have to do to keep things organized and easy to navigate.
Gestures & Spatial Model
Some parts of GNOME Shell have a clear, simple spatial model which makes navigation feel effortless. For example, workspaces are arranged in a vertical row, which you can cycle through via touchpad gestures or directional keyboard shortcuts.
However, this isn’t the case for everything, which is why parts of the shell can feel disconnected and difficult to navigate. A good example of this is the app grid: The first page opens with a “swarm” animation from the app grid icon, but then additional pages appear from outside the viewport.
Ideally the shell should have a clear, unified spatial model, so you always instinctively know where things are (even when they’re off-screen), and how to get to them, across input methods and form factors.
In order to enable that we need a more comprehensive set of gestures for navigating the core parts of the shell, as well as more intuitive and discoverable keyboard shortcuts. The same thing largely applies to touchscreens as well. Some gestures there are quite nice, but others are not “stick-to-finger”, or physically difficult to perform (e.g. pinch to open the overview).
One of the nice things about having a simpler spatial model and directional gestures to navigate the shell is that it also helps keyboard navigation. This is because it allows for directional shortcuts (e.g. Super + PgUp to move to the workspace above) which are more intuitive than non-directional ones (e.g. Super + M to open the notification popover).
Notifications & Agenda
The calendar popover contains a lot of different features (notifications, calendar, world clocks, weather). This can result in it feeling a bit large and unwieldy. Seeing as not everyone uses all the features in the popover we’re interested in giving people a way to customize which ones they want.
Ideally we’d like the features in this popover to be grouped in a more semantic way and easier to navigate to, especially with touchpads and touchscreens.
In addition to structural changes, we also want to make the notifications themselves better, for example by making notification actions accessible in the popover and grouping notifications by app. Mariana Pícolo is working on some of this as part of this year’s GSoC.
System Status Area
The system status area works well for the most part, but there are some parts we’d like to improve. For example, taking actions on individual items (e.g. toggling Bluetooth) always requires expanding the item’s sub-menu. Other items (e.g. Night Light) are only available in the menu when they’re enabled, which can feel unpredictable.
Our current search works well overall, but there are still areas where we think it could be better.
For the most part, search is currently just a portal to things inside apps (e.g. files, settings panels, contacts). However, we’ve found that search is often most interesting when it gives you something directly–launching an app, performing a calculation, looking up the weather, etc.
Having more such integrations and richer ways to interact with content from search is something we’re interested in exploring, along with generally expanding the kinds of things you can search for, e.g. your currently open windows.
Based on the above analysis, here are some tentative high-level goals for where we want to take the shell over the next few years:
Single entry point
Feels quick & easy to navigate
Structurally and visually elegant
In order to achieve these goals, these are some more concrete things we want to focus over the next few cycles (the previous blog post discusses these plans in more detail):
Better onboarding and bootup experience
Easier app launching
Simpler, unified spatial model
Better touchpad and touchscreen gestures
More medium-term, we’d also like to look at these:
More organic workspace workflows
Streamlined window management
Remember open apps across reboots
Improved system status area and search
Of course, some of these goals are much easier to achieve than others. Some of them may take years of work and multiple iterations to get right. That said, we think it’s important not to think about these ideas in isolation, but as part of a more comprehensive vision for the future.
The previous blog post lays out a fairly clear roadmap for the first few things we want to address. As that work becomes more concrete, we look forward to sharing and discussing it with the wider community. We hope this post will help provide context to that discussion.
These are exciting times, and we can’t wait to share more. Stay tuned!
A bit later than usual, but nonetheless here the changes that happened during April on GNOME Shell and Mutter.
The command-line extensions tool received a round of improvements, and now reports extension errors better. Switching the scale of a monitor now should update all interface elements properly on GNOME Shell. Another quality of life improvement that landed was the inclusion of ASCII alternatives in the search index, which for example allows “eteindre” match “éteindre” (French for “power off”).
GNOME Shell now integrates with the parental controls technology being developed across the GNOME stack. If there are user restrictions in place, GNOME Shell now filters the applications that are not supposed to be used by the particular user.
One important improvement that landed during April is the rewrite of GNOME Shell’s calendar daemon. This updated version should prevent a lot of heavy background processing of events. Given the extents of the improvements, this is being considered for backporting to GNOME 3.36, but the size of the changes are also considerable. Testing and validation would be appreciated.
April then ended with the release of both GNOME Shell 3.36.2 as well as 3.37.1.
The most prominent addition to Mutter during April was the introduction of Wayland fullscreen unredirect. This code has been under review for many months, and required coordination between different parts of the stack, to work properly. Unfortunately, because it requires a very recent version of Xwayland (1.20.8) containing the fixes necessary for it to work properly, it is not suitable for backporting.
Improvements to the screencasting code landed in preparation for further improvements to GNOME Shell’s built-in screen recorder. We hope to be able to have a single code path for capturing the screen contents, regardless of whether the consumer is the Desktop portal, or the built-in recorder.
Also an issue many users had ran into where the Super key did not work as it should when using multiple keyboard layouts in the X11 session was fixed!. A handful of other bug fixes and improvements was made for the GNOME 3.36 stable branch was also included in the 3.36.2 release in the end of the month.
Like GNOME Shell, April ended with Mutter’s 3.37.1 release as well.
With GNOME 3.36 out the door, it’s time to start thinking about what comes next. Those of us who work on GNOME UX are really proud of the latest GNOME shell release. It includes some major updates, particular the new lock screen and updated visuals, as well as a host of smaller improvements. 3.36 feels really nice to use, and has got a great response.
The lock screen work that we landed in 3.36 was the outcome of a long-running programme of UX work, which we first put together at the GNOME UX hackfest in London, back in 2017. There are still some outstanding pieces of the login/unlock experience that need to be filled in, and this is something that we hope to work on over the coming development cycle. However, we are also turning our attention to other aspects of the shell, which we have wanted to update for some time.
In the rest of this post, I’ll describe some of the areas that we’re hoping to improve, before going on to talk about how we’re going to do it.
The areas that we are looking at mainly relate to the overview, although in places they touch other areas of the experience. They are:
One thing we’ve wanted to improve for some time is the reliance on the alphabetical grid of launchers. This alphabetical organisation is mostly useful for finding applications by name, rather than presenting the apps that are most important or most likely to be used. This is something that the grid of frequent apps attempts to compensate for, but the overall experience doesn’t fit together as well as we’d like.
For application launching, the main goal is to make the apps space more useful – it needs to be easier to find the app that you want to launch. However, we also want to make it a more engaging and personal part of the experience, so the apps space feels like it belongs to you.
Overview spatial organization
We are all familiar with the organisation of the overview: window thumbnails in the middle, dash on the left, search up top, workspaces on the right. Functionally this works fairly well. However, the layout could be better in terms of how the pieces fit together. In particular, the arrangement of the different elements could do a better job of communicating how they relate to one another, especially in terms of navigation.
Boot and empty states
Boot into GNOME 3 and you’re presented with an empty desktop. Open the Activities Overview, and you’re again presented with an empty space in the middle of the screen. This experience isn’t particularly helpful, and doesn’t draw or guide the user into the experience.
We’ve been aware of these disadvantages of the current design for almost the entire history of GNOME 3, and we have experimented with a few different solutions, but never managed to get them to a usable state. Now, as we take another look at how the overview is arranged, we’d like to have another attempt at getting this right.
Right now, the touchpad gestures to move between workspaces are fairly straightforward: 4 finger swipe up and down. However, we currently don’t have an easy gesture to go in and out of the overview and, once you’re in it, we don’t have an easy way to navigate between the different areas (app launching and search in particular).
Being able to scoot around different areas of the desktop using a touchpad (or, indeed, a touchscreen) would be a big win and is something we are keen to allow. In order to do this, we need a simple model that people can use to navigate around, rather than having to learn multiple sets of gestures.
How we’re going to do it
Work in this area is already ongoing, and we’ve put a lot of thought into how to organise it in order to achieve a good result.
Research & design
Some of the functionality that is under review is a prominent part of GNOME 3, so it’s important that any changes we make are a genuine improvement. When it’s a core part of the desktop, even a small regression can be a significant issue for some users.
User research is going to be a key part of this process, both in order to ensure that we have a good understanding of user needs and requirements, as well as to test the outcome of design and development work, and modify it as necessary.
We have already done some initial, limited research, to find out how many windows, apps and workspaces are typically in use. The next research phase is currently being planned, and will involve a series of show and tell exercises, combined with semi-structured interviews, to get a better sense of how people use their desktops, and how the design can be improved for them.
Looking further ahead, we’ll conduct testing on any changes that we make, in order to evaluate how successful they are and ensure that users experience them as a genuine improvement over what came before.
At each stage, we hope to share the results of our findings.
Testing and iteration
Wherever possible, we are planning on landing UI changes incrementally, with an emphasis on maintaining a releasable product. This will allow us to pace our work and do testing and refinement throughout the development cycle, rather than just at the end.
When it isn’t possible to compartmentalise UI changes, we are planning on making development builds available early, in order to carry out testing and iteration over a longer period. This is likely to be the case for the bulk of any changes, due to the interconnected nature of the overview.
Watch this space
Design changes to GNOME Shell can generate both speculation and uncertainty. We’d like to mitigate this as much as we can, so that people understand what changes are coming, why they are being made, and why they can be confident that they are a real improvement.
Currently, there are a variety of experimental designs. However, we haven’t settled on a single proposal and don’t want to create false expectations by presenting them at this early stage. Hopefully once we have done more rounds of research we will be in a position to do this, and give a better idea of what UI changes might be coming further down the line. Until then, we ask you to be patient!
We also hope that the research, testing and feedback opportunities that we are planning will provide reassurance that any changes we eventually make will be positive. We are committed to make changes based on these data gathering exercises, if it turns out that the designs don’t perform as well as we’d hoped.
We will endeavour to provide progress reports on this work as it progreses, so watch this space for news!
During March, GNOME Shell and Mutter saw their 3.36.0 and 3.36.1 releases, and the beginning of the 3.38 development cycle. We’ve focused most of the development efforts on fixing bugs before starting the new development cycle.
From the development perspective, the 3.36.0 release was fantastic, and the number of regressions relative to the massive amount of changes that happened during the last cycle was remarkably small.
GNOME Shell saw continued improvements in its new Extensions app. New APIs were added to the Shell, which allows moving the Extensions app away into its own codebase. It also allows Shell to expose fewer interfaces through D-Bus. The Extensions app is now available on Flathub.
A number of other small bugs and crashes were fixed for 3.36.1. Notably, the blur effect now works properly with fractional scaling.
Following the 3.36.0 release, Mutter received various fixes to window streaming support. In contrast to streaming entire monitors, which was working properly, window streaming had a few quirks and misbehaviors. For 3.36.1, we’ve tracked down many issues around it and fixed them. Streaming windows is also done using DMA buffer sharing mechanisms.
On Wayland, sometimes new windows would use the wrong position to animate, leading to the zoom in animation look broken. This issue was fixed as well. Pasting images from Firefox does not freeze apps, specially Xwayland apps, anymore. We also fixed a series of bugs where Xwayland windows would show a black border when resizing.
Mutter now properly handles hardware cursors when hotplugging GPUs, and cursor hotspots now work correctly again on virtual machines. Sometimes cursors would rotate wrongly when on already rotated displays, and this was also fixed.
On the X11 front, Mutter now respects manually configured RandR panning on X11, and a bug preventing the correct monitor scale from being applied on X11 was also fixed.
Mutter now also finally respects the “middle mouse button” emulation setting exposed via GSettings.