System76: A Case Study on How Not To Collaborate With Upstream

Preface: the following post was written in the context of the events that happened in September. Some time has passed, and I held off on publishing in the hopes we could reach a happy ending with System76. As time has passed, that hope has faded. Attempts to reach out to System76 have not been productive, and I feel we’ve let the impression they’ve given the wider tech community about GNOME sit for far too long.  Some things have changed since I originally wrote the post, so some bits have been removed.

---

Recently there’s been some heated discussion regarding GNOME’s future. This has led to a lot of fear, uncertainty, and doubt being spread about GNOME, as well as attacks and hostility toward GNOME as a whole and toward individual contributors. This largely started due to the actions of one company’s employees in particular: System76.

This is not the first time System76 has been at the center of a public conflict with the GNOME community, nor is it the first time it was handled poorly. At this point, I no longer feel comfortable working with System76 without some sort of acknowledgment and apology for their poor behavior, and a promise that this won’t happen again.

You might be thinking: what sort of behavior are you talking about? What has System76 done to deserve this treatment? Well, it’s not any one incident – it’s a pattern of behavior that’s repeated multiple times over the past few years. I’ll share incidents I know of from the past, what their behavior has been like in the present, and my own thoughts on the future.

Disclaimer: The following content is my opinion and my opinion alone. I do not speak for GNOME as a whole, only for myself using the observations I’ve made. These observations may be incorrect.

System76 & the LVFS

This is the first major issue that I’m aware of. It will also showcase some of the behaviors that will repeat. Back in 2017 and 2018, Richard Hughes was communicating with System76 to get their firmware working on the LVFS and fwupd. Then in a sudden move, System76 announced their own infrastructure and software for firmware updates. They also told Richard that he should use their infrastructure instead of his. Richard expressed his surprise and disappointment at this move on his blog.

Initially this seems like a case of a communication breakdown alongside a technical disagreement. However, the real unacceptable behavior comes with System76’s response to Richard’s post. They start by sharing part of the correspondence between both parties. I believe the communication breakdown occurs in this section, where Richard explains why their current methods will not work for the LVFS. Then they go further and imply that the LVFS is handing away private data:

Be wary

We had intended to use LVFS in the future, but that is no longer the case; there are too many problems with the project.
If you want to use LVFS, disable the data collection. There’s no need for it. Understand that the first instinct of the project leaders was to unnecessarily opt-in data collection from user installations.
Understand that you’re electing for your distribution to communicate with third party servers. The more servers your distribution communicates with out of the box (especially as root), the more surface area there is for vulnerabilities. Heavily scrutinize any addition of a third-party server for updates.
Understand that if you are a company specializing in Linux-only devices and considering using LVFS, you are handing your private sales data to LVFS. We suggest hosting LVFS on your own servers.

System76 employees continued to spread this idea into early 2019 online, likely prompting a post from Richard to refute their claims. Later on the LVFS team put in work to support the use case of System76 so that vendors could use the LVFS without any sort of reporting. Soon after System76 began using the LVFS and fwupd without any fanfare or retraction of their prior statements.

Now, before we move on to the next incident there are a few key takeaways that should be kept in mind. These will be repeating behaviors:

• System76 blindsided upstream with criticism and advertised their own in-house solutions.
• System76 spread misinformation for months after not getting their way.

System76, Ubuntu, & Bug Fixes

In 2019 Sebastien Bacher from Canonical shared a post about System76’s treatment of upstreams – in this case, both Ubuntu and GNOME:

I saw a few cases of those situations happening recently

1. System76 / Pop! OS finds a bug (where ‘find’ often means that they confirm an existing upstream bug is impacting their OS version)

2. They write a patch or workaround, include it in their package but don’t upstream the change/fix (or just drop a .patch labelled as workaround in a comment rather than submitting it for proper review)

3. Later-on they start commenting on the upstream (Ubuntu, GNOME, …) bugs trackers, pointing out to users that the issue has been addressed in Pop! OS, advertising how they care about users and that’s why they got the problem solved in their OSSystem76 / Pop! OS team, while you should be proud of the work you do for you users I think you are going the wrong way there. Working on fixes and including them early in your product is one thing, not upstreaming those fixes and using that for marketing you as better than your upstreams is a risky game.

At this point in time System76 had already established a pattern of behavior where instead of working with upstreams, they take the opportunity to market themselves and their solutions.

System76, GNOME Shell, & Tiling

This is a fairly straightforward incident, but it’s one that’s concerning as it involves a misleading narrative that System76 has been running for two years. In late 2019 System76 began work on an extension that allowed for i3-like tiling in GNOME. When an upstream designer reached out for collaboration on tiling upstream Jeremy Soller, principal engineer at System76, refused to work with him. Now, the true problematic part is that System76 has continued to repeat that upstream is not interested in tiling for the past few years. A few examples:

I’m afraid GNOME does not want to support quarter tiling windows.

It would be nice if GNOME wanted to have tiling window functionalities integrated into their desktop. But the truth is that they don’t.

– Michael Murphy

This is untrue, and the linked tweet shows that. Yet System76 employees keep repeating the idea that we don’t want tiling.

System76 & GNOME 40

Soon after we announced the direction for GNOME 40, Jeremy Soller surprised us with the statement that System76 did not “consent” to the new GNOME Shell design. He also claimed that feedback from their designer was dismissed. GNOME 40’s design was largely discussed on design calls that were not recorded, which makes this claim hard to verify. It’s also hard to address because while there may not have been any intent of harm, their designer may still have felt unheard or hurt – causing pain doesn’t require intending to cause pain. So, I will focus on what I can address.

Jeremy claims and has continued to claim that their proposals were rejected. That is not the case. The reality is that System76 had a designer involved for roughly one year of the three-year design process. Their designer participated in design calls and discussions with the design team, and her role was focused on the UX research itself (defining research questions, running interviews, parsing the results, etc.). There were never any mockups or concrete proposals from System76 during the design process. I’ve also heard that System76 conducted a survey and did not share the results with the team working on the 40 design.

To my understanding, the one time System76 made a proposal was at the very end of the design process during a meeting the design team had with stakeholders. There they pitched COSMIC and didn’t receive the feedback they wanted at the time. Perhaps this is what they were referring to, but at this point it was too late for any proposed changes to be made.

The behavior of the company repeated the initial pattern. When System76 did not get their way with their design, they started sharing misinformation. This particular instance of misinformation has been incredibly harmful for GNOME’s reputation. I’ve often seen the first two tweets brought up as “proof” that GNOME doesn’t want to work with downstreams or listen to users. In reality they are an example of how System76 acts toward others when they don’t get what they want.

System76, libadwaita, & GTK Themes

Now we’ve arrived at the latest incident. I’ve been involved in this one more than the others, and I have watched it unfold from the beginning to the present. I’ve decided to go into further depth, breaking down the most important parts by the day.

September 1

This all started at the beginning of the month, when Jeremy noticed that we override gtk-theme-name and set the Adwaita stylesheet for apps using libadwaita. He shared this on his Twitter. Jeremy then posted misinformation about dark style support. Why is this misinformation?

• Alexander had introduced a PR for a global dark style preference that would integrate it into the upstream platform a few days prior
• Jeremy was actually aware of this at the time

Jeremy then very publicly called out GNOME as a whole – something that a few people interpreted as a threatening ultimatum. Jeremy was soon made aware that libadwaita is a WIP library and that there is still time to work with us regarding vendor theming. Yet, this was not the end for him posting about this on Twitter. Federico wrote and linked a blog post detailing our goals and what can be done to help resolve the situation, with help from Alexander.

September 2

Jeremy replied to Federico’s blog post with a quote from Alexander. This quote was taken out of context and assumed a different meaning, a meaning that is convenient to the narrative that was building up until this point. In reality Alexander was only explaining the status of things as they are in the present, and what the current goals are. Jeremy also quoted an old blog post from Adrien from before libadwaita existed. Do note that in said blog post Adrien went over the possibility for vendors and users to style within this framework despite his personal opinion. When asked to work with us upstream instead of going to Twitter, Jeremy refused – he would not work with us unless he got exactly what he wanted.

September 3

Jeremy stated that System76 was excited to use libadwaita. The very next day Michael Murphy, one of their core team members, contradicted this saying:

I wasn’t even aware that libadwaita even existed when Jeremy started tweeting about it. I don’t think any of us knew about libadwaita and it’s goals.

Michael also made the claim that we don’t want to collaborate when at this point we’d asked them to collaborate instead of pointing fingers multiple times, as shown in the prior archives.

Finally after three days an opportunity appeared to talk and I took it. Below is a summary of the conversation, of which I also kept screenshots of:

• Jeremy sent the proposal he mentioned
• I read through the proposal, then gave my feedback. I said I thought that it was interesting, but I also gave a detailed breakdown on why I didn’t think it would be implemented, including the following points:
  • The GTK team wants to put platform libraries in charge of platform styling
  • Most app developers I know (even those who hadn’t signed the well-known open letter at the time) are not happy with the state of arbitrarily applied stylesheet changes
  • Many app developers don’t want to take on the burden of supporting multiple stylesheets
  • Other stylesheets may not be up-to-date implementing Adwaita’s API, re-creating the existing problem
  • “Hard coding” Adwaita is part of our work toward things like a recoloring API for apps and a properly supported dark style preference
• I expressed there was room to discuss distro branding of default apps, but not a will to bring back arbitrary stylesheet support.
• I explained the concerns of app developers regarding custom widgets
• Jeremy expressed his fear about replacing default apps because of these changes
• I explained that it would be better to work with us on a non-stylesheet method for vendor branding
• As the conversation wrapped up, I asked that in the future they take issues to GitLab before going to Twitter. I also expressed the pain they had caused and subtly tried to tell them that some developers would want an apology before working with them.
• Jeremy did not acknowledge the first part and refused to make an apology.

Jeremy then opened the GTK issue, and was redirected to libadwaita. After some discussion on the libadwaita issue he opened a merge request.

September 6

After a lot of discussion on the issue and the MR, the merge request was closed once was made clear that:

• Third-party and core application developers largely did not want the proposed solution.
• Multiple developers, including myself, were not comfortable working with System76 without an acknowledgment of their behavior and an apology.

The Aftermath

Once again, System76 employees decided to spread misinformation and complaints on Twitter instead of making an attempt to work with us upstream on their issues. It’s reasonable to assume that they learned about libadwaita and began posting ultimatums on Twitter on the same day. Once again, they have spread misinformation about the project and our goals.

In the weeks since the MR was closed, a few different employees have continued spreading misinformation on Twitter about the situation. Their existing misinformation has already caused ripples within the community, getting many people stirred up and making claims like GTK4 being only for GNOME, or GNOME wanting to be the “only” desktop on Linux because of the approach to themes we have. We’ve had a lot of vitriol thrown our way.

System76’s words also have reach beyond the Linux ecosystem, and large content creators are only hearing their side of the story.

There have been additional attempts by others to reach out diplomatically, but those seem to have fallen through. Just the other day the CEO claimed that there were no technical or quality reasons for the change, heavily implying that it was done just to hurt downstreams.

With all of the misinformation and the consequences of said misinformation, and with the refusal to listen or engage with our own needs, I do not feel like it is worth my time to engage with System76. Until they recognize their behavior for what it is and make a commitment to stop it, I will not be doing free labor for them. If they can do that, I would be happy to work with them on resolving their issues.

Epilogue: What’s Really Going On With Theming?

Now that we’ve addressed System76’s hand in stirring up controversy, let me clarify what the situation actually is, and what vendors and other desktop environments can do moving forward.

Terms To Know

This is a high-level overview, but since we’re talking about technical details there will be some terms to know:

• Stylesheet: Compilations of CSS styles for widgets. “Themes” as we know them are stylesheets, e.g. Adwaita, Arc, Pop, or Yaru.
• Platform library: A platform library is a collection of widgets, useful functions, constants, and styles for apps on a platform to use. Granite and libadwaita are examples of platform libraries.
• Visual API: An API or “contract” for the visuals of an application.

GTK & Theming

Let me clear this up immediately: nothing about the theming infrastructure has changed in GTK itself between GTK3 and GTK4. The only changes are listed in the migration guide, and are mostly removed or replaced GTK-specific CSS functions. All the changes that have caused controversy have to do with libadwaita.

You might have seen some people bring up two particular issues as counterpoints to this:

• Consider removing the gtk-theme-name setting
• Remove the prefer-dark setting

What do those two issues mean? Well, it means that the GTK team thinks that these particular issues are better handled in platform libraries. This does not mean only libadwaita or only libgranite, nor does it mean that GTK-based applications would no longer be able to have a dark theme or different themes. The platform library would be responsible for handling those items in accordance with their goals.

Desktops X, Y, and Z could work together and create a libxyz that loads different stylesheets just like GTK does now, or they could each make their own platform library that handles their specific needs. The proposed GTK changes would simply put the choice in the hands of each platform.

GNOME, libadwaita, & Theming

In April when Adwaita was moved to libadwaita, the controversial change to override gtk-theme-name was made. The commit explains the reasoning for this particular change:

Ensure Adwaita is loaded instead of the default, take care to load Adwaita-hc when HighContrast is selected system-wide.

Why do we want to ensure libadwaita is loaded at all instead of other stylesheets? It comes down to two key things:

• Adwaita for platform integration
• Adwaita as a visual API

In GNOME, we want to have a stable platform for app developers to use to build new, useful, and well-integrated apps. Part of having a stable platform is having a stable visual API. The GNOME Human Interface Guidelines lay out the ideas and goals for apps targeting GNOME to follow, and libadwaita helps developers implement those ideas. The stylesheet itself is part of keeping interfaces in line with the HIG.

With libadwaita the widgets, the stylesheet, and the HIG are all tied together in one easy to use package. With each new version of GNOME we can iterate on them all at once so it will be easier for developers to keep up with different changes. This is in contrast to the current state of development where app developers need to implement their own versions of common patterns or copy-paste them from other apps.

Adwaita being used this way means that apps also need to rely on it being there, and apps will make assumptions based on that. For example, we want it to be very simple for app developers to recolor widgets. Before if apps wanted to do this, they needed to depend on sassc and libsass and occasionally regenerate the themes on-the-fly. Now all they need to do is set the background-color on a widget.

This new method works perfectly with the new Adwaita stylesheet. Here we have a simple Color Picker app:

Recoloring still works well with stylesheets that are based on Adwaita, like Yaru:

Things begin breaking down with stylesheets that deviate too far from Adwaita:

Without enforcing Adwaita as the stylesheet apps may not look correct or in some cases may not function at all. For a more in-depth breakdown of the problem with more examples, see Tobias Bernard’s post “Restyling apps at scale.”