So I have spoken about what is our vision for Fedora Workstation quite a few times before, but I feel it is often useful to get back to it as we progress with our overall effort.So if you read some of my blog posts about Fedora Workstation over the last 5 years, be aware that there is probably little new in here for you. If you haven’t read them however this is hopefully a useful primer on what we are trying to achieve with Fedora Workstation.
The first few years after we launched Fedora Workstation in 2014 we focused on lot on establishing a good culture around what we where doing with Fedora, making sure that it was a good day to day desktop driver for people, and not just a great place to develop the operating system itself. I think it was Fedora Project Lead Matthew Miller who phrased it very well when he said that we want to be Leading Edge, not Bleeding Edge. We also took a good look at the operating system from an overall stance and tried to map out where Linux tended to fall short as a desktop operating system and also tried to ask ourselves what our core audience would and should be. We refocused our efforts on being a great Operating System for all kinds of developers, but I think it is fair to say that we decided that was to narrow a wording as our efforts are truly to reach makers of all kinds like graphics artists and musicians, in addition to coders. So I thought I go through our key pillar efforts and talk about where they are at and where they are going.
One of the first things we concluded was that our story for people who wanted to deploy applications to our platform was really bad. The main challenge was that the platform was moving very fast and it was a big overhead for application developers to keep on top of the changes. In addition to that, since the Linux desktop is so fragmented, the application developers would have to deal with the fact that there was 20 different variants of this platform, all moving at a different pace. The way Linux applications was packaged, with each dependency being packaged independently of the application created pains on both sides, for the application developer it means the world kept moving underneath them with limited control and for the distributions it meant packaging pains as different applications who all depended on the same library might work or fail with different versions of a given library. So we concluded we needed a system which allowed us to decouple of application from the host OS to let application developers update their platform at a pace of their own choosing and at the same time unify the platform in the sense that the application should be able to run without problems on the latest Fedora releases, the latest RHEL releases or the latest versions of any other distribution out there. As we looked at it we realized there was some security downsides compared to the existing model, since the Os vendor would not be in charge of keeping all libraries up to date and secure, so sandboxing the applications ended up a critical requirement. At the time Alexander Larsson was working on bringing Docker to RHEL and Fedora so we tasked him with designing the new application model. The initial idea was to see if we could adjust Docker containers to the desktop usecase, but Docker containers as it stood at that time were very unsuited for the purpose of hosting desktop applications and our experience working with the docker upstream at the time was that they where not very welcoming to our contributions. So in light of how major the changes we would need to implement and the unlikelyhood of getting them accepted upstream, Alex started on what would become Flatpak. Another major technology that was coincidentally being developed at the same time was OSTree by Colin Walters. To this day I think the best description of OSTree is that it functions as a git for binaries, meaning it allows you a simple way to maintain and update your binary applications with minimally sized updates. It also provides some disk deduplication which we felt was important due to the duplication of libraries and so on that containers bring with them. Finally another major design decision Alex did was that the runtime/baseimage should be hosted outside the container, so make possible to update the runtime independently of the application with relevant security updates etc.
Today there is a thriving community around Flatpaks, with the center of activity being flathub, the Flatpak application repository. In Fedora Workstation 35 you should start seeing Flatpak from Flathub being offered as long as you have 3rd party repositories enabled. Also underway is Owen Taylor leading our efforts of integrating Flatpak building into the internal tools we use at Red Hat for putting RHEL together, with the goal of switching over to Flatpaks as our primary application delivery method for desktop applications in RHEL and to help us bridge the Fedora and RHEL application ecosystem.
You can follow the latest news from Flatpak through the official Flatpak twitter account.
So another major issue we decided needing improvements was that of OS upgrades (as opposed to application updates). The model pursued by Linux distros since their inception is one of shipping their OS as a large collection of independently packaged libraries. This setup is inherently fragile and requires a lot of quality engineering and testing to avoid problems, but even then sometimes things sometimes fail, especially in a fast moving OS like Fedora. A lot of configuration changes and updates has traditionally been done through scripts and similar, making rollback to an older version in cases where there is a problem also very challenging. Adventurous developers could also have done changes to their own copy of the OS that would break the upgrade later on. So thanks to all the great efforts to test and verify upgrades they usually go well for most users, but we wanted something even more sturdy. So the idea came up to move to a image based OS model, similar to what people had gotten used to on their phones. And OSTree once again became the technology we choose to do this, especially considering it was being used in Red Hat first foray into image based operating systems for servers (the server effort later got rolled into CoreOS as part of Red Hat acquiring CoreOS). The idea is that you ship the core operating system as a singular image and then to upgrade you just replace that image with a new image, and thus the risks of problems are greatly reduced. On top of that each of those images can be tested and verified as a whole by your QE and test teams. Of course we realized that a subset of people would still want to be able to tweak their OS, but once again OSTree came to our rescue as it allows developers to layer further RPMS on top of the OS image, including replacing current system libraries with for instance newer ones. The great thing about OSTree layering is that once you are done testing/using the layers RPMS you can with a very simple command just drop them again and go back to the upstream image. So combined with applications being shipped as Flatpaks this would create an OS that is a lot more sturdy, secure and simple to update and with a lot lower chance of an OS update breaking any of your applications. On top of that OSTree allows us to do easy OS rollbacks, so if the latest update somehow don’t work for you can you quickly rollback while waiting for the issue you are having to be fixed upstream. And hence Fedora Silverblue was born as the vehicle for us to develop and evolve an image based desktop operating system.
You can follow our efforts around Silverblue through the offical Silverblue twitter account.
So Flatpak helped us address a lot of the the gaps for making a better desktop OS on the application side and Silverblue was the vehicle for our vision on the OS side, but we realized that we also needed some way for all kinds of developers to be able to easily take advantage of the great resource that is the Fedora RPM package universe and the wider tools universe out there. We needed something that provided people with a great terminal experience. We had already been working on various smaller improvements to the terminal for a while, but we realized we needed something a lot more substantial. Accessing an immutable OS like Silverblue through a terminal window tends to be quite limiting. So that it is usually not want you want to do and also you don’t want to rely on the OSTree layering for running all your development tools and so on as that is going to be potentially painful when you upgrade your OS.
Luckily the container revolution happening in the Linux world pointed us to the solution here too, as while containers were rolled out the concept of ‘pet containers’ were also born. The idea of a pet container is that unlike general containers (sometimes refer to as cattle containers) pet container are containers that you care about on an individual level, like your personal development environment. In fact pet containers even improves on how we used to do things as they allow you to very easily maintain different environments for different projects. So for instance if you have two projects, hosted in two separate pet containers, where the two project depends on two different versions of python, then containers make that simple as it ensures that there is no risk of one of your projects ‘contaminating’ the others with its dependencies, yet at the same time allow you to grab RPMS or other kind of packages from upstream resources and install them in your container. In fact while inside your pet container the world feels a lot like it always has when on the linux command line. Thanks to the great effort of Dan Walsh and his team we had a growing number of easy to use container tools available to us, like podman. Podman is developed with the primary usecase being for running and deploying your containers at scale, managed by OpenShift and Kubernetes. But it also gave us the foundation we needed for Debarshi Ray to kicked of the Toolbx project to ensure that we had an easy to use tool for creating and managing pet containers. As a bonus Toolbx allows us to achieve another important goal, to allow Fedora Workstation users to develop applications against RHEL in a simple and straightforward manner, because Toolbx allows you to create RHEL containers just as easy as it allows you to create Fedora containers.
You can follow our efforts around Toolbox on the official Toolbox twitter account
Ok, so between Flatpak, Silverblue and Toolbox we have the vision clear for how to create a robust OS, with a great story for application developers to maintain and deliver applications for it, to Toolbox providing a great developer story on top of this OS. But we also looked at the technical state of the Linux desktop and realized that there where some serious deficits we needed to address. One of the first one we saw was the state of graphics where X.org had served us well for many decades, but its age was showing and adding new features as they came in was becoming more and more painful. Kristian Høgsberg had started work on an alternative to X while still at Red Hat called Wayland, an effort he and a team of engineers where pushing forward at Intel. There was a general agreement in the wider community that Wayland was the way forward, but apart from Intel there was little serious development effort being put into moving it forward. On top of that, Canonical at the time had decided to go off on their own and develop their own alternative architecture in competition with X.org and Wayland. So as we were seeing a lot of things happening in the graphics space horizon, like HiDPI, and also we where getting requests to come up with a way to make Linux desktops more secure, we decided to team up with Intel and get Wayland into a truly usable state on the desktop. So we put many of our top developers, like Olivier Fourdan, Adam Jackson and Jonas Ådahl, on working on maturing Wayland as quickly as possible.
As things would have it we also ended up getting a lot of collaboration and development help coming in from the embedded sector, where companies such as Collabora was helping to deploy systems with Wayland onto various kinds of embedded devices and contributing fixes and improvements back up to Wayland (and Weston). To be honest I have to admit we did not fully appreciate what a herculean task it would end up being getting Wayland production ready for the desktop and it took us quite a few Fedora releases before we decided it was ready to go. As you might imagine dealing with 30 years of technical debt is no easy thing to pay down and while we kept moving forward at a steady pace there always seemed to be a new batch of issues to be resolved, but we managed to do so, not just by maturing Wayland, but also by porting major applications such as Martin Stransky porting Firefox, and Caolan McNamara porting LibreOffice over to Wayland. At the end of the day I think what saw us through to success was the incredible collaboration happening upstream between a large host of individual contributors, companies and having the support of the X.org community. And even when we had the whole thing put together there where still practical issues to overcome, like how we had to keep defaulting to X.org in Fedora when people installed the binary NVidia driver because that driver did not work with XWayland, the X backwards compatibility layer in Wayland. Luckily that is now in the process of becoming a thing of the past with the latest NVidia driver updates support XWayland and us working closely with NVidia to ensure driver and windowing stack works well.
So now we had a clear vision for the OS and a much improved and much more secure graphics stack in the form of Wayland, but we realized that all the new security features brought in by Flatpak and Wayland also made certain things like desktop capturing/remoting and web camera access a lot harder. Security is great and critical, but just like the old joke about the most secure computer being the one that is turned off, we realized that we needed to make sure these things kept working, but in a secure and better manner. Thankfully we have GStreamer co-creator Wim Taymans on the team and he thought he could come up with a pulseaudio equivalent for video that would allow us to offer screen capture and webcam access in a convenient and secure manner.
As Wim where prototyping what we called PulseVideo at the time we also started discussing the state of audio on Linux. Wim had contributed to PulseAudio to add a security layer to it, to make for instance it harder for a rogue application to eavesdrop on you using your microphone, but since it was not part of the original design it wasn’t a great solution. At the same time we talked about how our vision for Fedora Workstation was to make it the natural home for all kind of makers, which included musicians, but how the separateness of the pro-audio community getting in the way of that, especially due to the uneasy co-existence of PulseAudio on the consumer side and Jack for the pro-audio side. As part of his development effort Wim came to the conclusion that he code make the core logic of his new project so fast and versatile that it should be able to deal with the low latency requirements of the pro-audio community and also serve its purpose well on the consumer audio and video side. Having audio and video in one shared system would also be an improvement for us in terms of dealing with combined audio and video sources as guaranteeing audio video sync for instance had often been a challenge in the past. So Wims effort evolved into what we today call PipeWire and which I am going to be brave enough to say has been one of the most successful launches of a major new linux system component we ever done. Replacing two old sound servers while at the same time adding video support is no small feat, but Wim is working very hard on fixing bugs as quickly as they come in and ensure users have a great experience with PipeWire. And at the same time we are very happy that PipeWire now provides us with the ability of offering musicians and sound engineers a new home in Fedora Workstation.
You can follow our efforts on PipeWire on the PipeWire twitter account.
Hardware support and firmware
In parallel with everything mentioned above we where looking at the hardware landscape surrounding desktop linux. One of the first things we realized was horribly broken was firmware support under Linux. More and more of the hardware smarts was being found in the firmware, yet the firmware access under Linux and the firmware update story was basically non-existent. As we where discussing this problem internally, Peter Jones who is our representative on UEFI standards committee, pointed out that we probably where better poised to actually do something about this problem than ever, since UEFI was causing the firmware update process on most laptops and workstations to become standardized. So we teamed Peter up with Richard Hughes and out of that collaboration fwupd and LVFS was born. And in the years since we launched that we gone from having next to no firmware available on Linux (and the little we had only available through painful processes like burning bootable CDs etc.) to now having a lot of hardware getting firmware update support and more getting added almost on a weekly basis.
For the latest and greatest news around LVFS the best source of information is Richard Hughes twitter account.
In parallel to this Adam Jackson worked on glvnd, which provided us with a way to have multiple OpenGL implementations on the same system. For those who has been using Linux for a while I am sure you remembers the pain of the NVidia driver and Mesa fighting over who provided OpenGL on your system as it was all tied to a specific .so name. There was a lot of hacks being used out there to deal with that situation, of varying degree of fragility, but with the advent of glvnd nobody has to care about that problem anymore.
We also decided that we needed to have a part of the team dedicated to looking at what was happening in the market and work on covering important gaps. And with gaps I mean fixing the things that keeps the hardware vendors from being able to properly support Linux, not writing drivers for them. Instead we have been working closely with Dell and Lenovo to ensure that their suppliers provide drivers for their hardware and when needed we work to provide a framework for them to plug their hardware into. This has lead to a series of small, but important improvements, like getting the fingerprint reader stack on Linux to a state where hardware vendors can actually support it, bringing Thunderbolt support to Linux through Bolt, support for high definition and gaming mice through the libratbag project, support in the Linux kernel for the new laptop privacy screen feature, improved power management support through the power profiles daemon and now recently hiring a dedicated engineer to get HDR support fully in place in Linux.
So to summarize. We are of course not over the finish line with our vision yet. Silverblue is a fantastic project, but we are not yet ready to declare it the official version of Fedora Workstation, mostly because we want to give the community more time to embrace the Flatpak application model and for developers to embrace the pet container model. Especially applications like IDEs that cross the boundary between being in their own Flatpak sandbox while also interacting with things in your pet container and calling out to system tools like gdb need more work, but Christian Hergert has already done great work solving the problem in GNOME Builder while Owen Taylor has put together support for using Visual Studio Code with pet containers. So hopefully the wider universe of IDEs will follow suit, in the meantime one would need to call them from the command line from inside the pet container.
The good thing here is that Flatpaks and Toolbox also works great on traditional Fedora Workstation, you can get the full benefit of both technologies even on a traditional distribution, so we can allow for a soft and easy transition.
So for anyone who made it this far, appoligies for this become a little novel, that was not my intention when I started writing it :)
Feel free to follow my personal twitter account for more general news and updates on what we are doing around Fedora Workstation.
I don’t quite understand how you went from looking for something that “allows us to do easy OS rollbacks” to “move to a image based OS model, similar to what people had gotten used to on their phones”. Isn’t an approach like Nix a more direct solution?
This is fantastic work — it it only the redhat team making this work.
I’d love to help push it forward and there’s an incredible amount of engineering talent out there happy to contribute. I hope all this incredible work can get momentum and let me know how I can contribute
Excellent summary! Very informative. Distributions should do more of this.
Why is this on GNOME’s website (not in Fedora’s blog)?
There is no meaningful distinction between Gnome and Red Hat at this point. Although they don’t employ every single gnome developer and gnome is technically open to outside contributions, they are both Gnome’s primary driver as far as paid development resources and are setting the path forward.
Gnome is Red Hat’s official GUI layer which you are more than welcome to incorporate into your distro so long as exactly what they want also happens to work for you. Basically its available in any color you like so long as its black.
None of that is true. The reason it shows up on GNOME’s blog system is because that’s where Christian elects to host his blog. The blog post shows up on both the Fedora Planet and the GNOME planet because the blog post is tagged accordingly and the blog is registered with both systems.
Yeah, as Neil says, my blog has been hosted on the GNOME blogroll for years, since before I joined Red Hat, and I have simply not bothered moving it both because its a hassle, but also because it would ‘reset’ my Google pagerank etc.
I’m not sure if this have come up in flatpak developer conversation before, but imho, one thing missing in flatpak right now is a way for developers to charge for software .. flatpak is already very close to APK, however the lack of a way for repositories to charge for software, (eg: like Red Hat subscription manager) means there’s less incentive to build more things for Linux ..
For games there’s Steam playing this role, but there’s no easy way to do it for non games .. it is also not easy to run a 3rd party subscription manager ..
It would be great if Gnome Software could let me add a game developer’s repository, then pay for one of their games, then install it as a flatpak like I can do with regular apps, complete with a the proton version it needs.
Yes, need this
So many words and no mention of GtkFileChooser! It’s not funny! Windows 95 had it better!
Great post, and kudos on the titanic work that you and your peers did.
One thing still missing for me is stable fractional scaling on Wayland, unfortunately it’s a dealbreaker when you have to use XWayland apps (e.g. java) for work, blurriness makes it unusable.
Silverblue talks to a problem redhat itself created. Most distributions use library versioning based package naming policies, starting with debian. This allows multiple abi versions to be installed and active “side by side”. Especially, for in-place upgrading, this means you can have old libraries still active for those packages which have not yet been updated while processing a mass upgrade. Even arch uses library versioning, although pretty much only for glibc and a few core libraries, which allows at least pacman and core utilities to be stable during in-place upgrades.
Incidentally, every other major RPM distro also does versioned library naming, particularly opensuse, which closely matches debian naming conventions. Up to two decades of wasted time with upgrade failures in Redhat/CentOS and then Fedora Core/Fedora could actually have been avoided if Redhat had simply adopted just this one practice.
The reason for Fedora’s style of library packaging is that it matches the philosophy of trying to encourage one version of a library in a distro and the latest version of that library. For staying close to upstream projects as Fedora does, that means working with upstreams to upgrade their stuff and move forward. The Debian/openSUSE model of packaging means that you don’t have that pressure because the old libraries stick around.
The philosophy is flawed and Flatpak underlines that. It’s library versioning in another form.
No it’s not. You are oversimplifying it. It’s moving the responsibility on the app provider and end user. But not entirely! There’s sandbox.
I’ve been using Fedora since the beginning but this is where we part company. I have no use for Silverblue at all. If this is your vision for Fedora I’m out.
a friend suggest you a Flatpaked app. And you.. “just this time”… I promise it.
Some weeks later you find another interesting app… If binary, you must compile from the source.. ouch! I’m late… I press accept and i leave, bb!
A year later. Your brand new i5 11600KF and your 32 Gigs Corsair can’t carry all garbage you accumulated inside this little bags, dozens of easy-packaging pieces of q system that. Is not UNIX (and much less, a LINUX OS).
If you want an OS for children, buy an Apple. Believeme, is just plug and play! Dont mind the unnecessary files camping on the Mobo, filling the RAM, and the cheap 240Gigs SSD they sell you (same for the not upgradable RAM crazy obsoletly cheap! Just pay for it, repair permissions once a month, and enjoy! (Enjoy as you can cause 3 years later it will devaluate a 85% with the apple Renove program).
I loved fedora, since 2002-03 I got my CD. I enjoyed all distros. But If canonical and IBM are dreaming with an packed word, as an Amazon dream with boxes (yes boxes) with wings… I must ask you for write correctly the docs and explain advantage(s) and issues of the appimage, snap, osx app, and selfcontained mcMenues. Please… Keep it away of my Arched OS’s .
25 years working in a agency with macs… Was enought… Thanks!
Btw… app is not KISS. anyway, kisses and reggards.
The apps in flatpaks are using bases, which share most of the libraries (GNOME, KDE, etc.) you have probably many versions of them installed or from different bases.
This is not a proper support channel, but you should ask on some advices how to clean up that.
This has been fixed in flatpak:
“Long story short, with the upcoming releases of Flatpak 1.10 and GNOME Software 40, both will remove unused EOL runtimes during update operations and uninstall operations, freeing up disk space for users. If you maintain a software manager that supports Flatpak, you may consider using the new API to ensure unused runtimes are regularly cleaned up”
When can we look forward to an easy way to secure the boot process in Fedora?
It is on the agenda as something we are looking at, but it is not a simple fix as some of the changes needed are some fundamental that they break certain things that people have relied on for decades, so often what might seem like a trivial fix requires Months or Years worth of effort to work on pre-requisites, especially if you have to deal with upstream projects who doesn’t agree with your security assesment.
I know it is not easy or trivial. But it is necessary. We need the foundation of Fedora solid before we need new features. Thanks for your insightful reply and your continued dedication to improving Linux.
You have perhaps not touched on a weak-spot “Documentation”. There needs to be some effort at eliminating old Fedora21 stuff therein, and bringing the documentation (Install guide, admin guide) current with Fedora 35 version.
I have rewritten part of the Fedora 34 installation guide, so that it has good grammar, is current. It is available. My wish is that the installation guide move to Flathub, and that a different application from asciidoc be used. By replacing asciidoc with LibreOffice’s writer, for example, one can immediately take advantage of the inline “track changes” feature, and the ability to have collaboration for accuracy and grammar. Collaboration will occur more readily from a larger audience, simply because most Linux and LibreOffice users are familiar with *.odt, *.docx formats. As a reminder, LibreOffice writer can produce PDF files as well as other formats.
Wow, thank you!!
I am overwhelmed at the amount of change taking place within the RHat organization in regard to Linux. It seems like the re engineering is being done by a cast of 50 full time developers or more. Am I understating the numbers?
Great post, I love reading these over time. It has many typos; I made https://gist.github.com/skierpage/3156f4eaf5b1ca9dc9ae4ab7b6a51478 instead of a big blog comment.
I’ve happily used Fedora KDE spin since F24. It just keeps getting better. I’m not sure how or when to jump to Silverblue from `dnf system-update`.
if you want “Silverblue” but with KDE instead, it would be called “Kinoite” which reached Beta a few days ago