There has been much talk about modularity recently. Fedora even has a working group on this topic. Modularity is such a generic term that it is a bit hard to figure out what this is all about, but the wiki page gives some hints: …base module… …docker image… …reduced dependencies… …tooling…
A very high-level reading would be that this is about defining tools and processes to deliver software in modules, which can be larger units than packages, and e.g. come in the form of container images.
Another reading would be that this is an effort to come to grips with the changing role of distributions in a world were containers are the dominant way to run and deploy applications.
Modularity on the desktop
In this post, I want to look at how this modularity effort looks from the desktop perspective, and what we are doing in this area.
Our main goal for a while has been to make it easier to get desktop applications from application developers to users.
In the traditional Linux distribution world, this is a total nightmare: Once you’ve written your application, you need to either learn how to create an Ubuntu .deb, an Arch .pkg, a Fedora .rpm, to name just a few, and then follow lengthy processes to get your packages accepted into these distributions.
And once you’ve done all that, you will get bug reports that your application is broken with the one or other version of one of your dependencies. If not right away, then a few months down the road when the distributions move on to the latest and greatest releases. So, keeping those packages working requires the constant attention of a package maintainer.
To improve on this sad state of affairs, we need to make applications much less dependent on the OS they run on, and on the libraries that happen to come with the OS.
At this point, you might say: Aha, you want to bundle dependencies! Yes, but read on. Bundling is a bad word in the traditional distribution world. It implies duplication, since multiple applications may bundle the same library. And having multiple copies of the same library (possibly different versions, too), makes it harder to ensure that bug and security fixes get applied to all the copies. Not to mention that all the duplication consumes bandwidth when you have to download it all.
Bundling everything with the application certainly has its drawbacks. But there are several things we can do to preserve most of the benefits of bundling, while avoiding the worst of the problems.
One takeaway from the modularity discussions mentioned earlier is that going to larger units than individual packages (ie modules) has advantages: we thin out the dependency graph, and we get a large piece of functionality that can be tested, installed and updated as a unit.
And we can use a smart storage and transport mechanism such as OSTree to limit the impact that duplication has on bandwidth.
Desktop applications commonly depend on similar sets of libraries: the GTK+ stack is a good example, kdelibs is another. Treating these common library stacks as modules makes a lot of sense. We decided to call these modules runtimes. Applications can declare a single dependency on the runtime they need, and get a well-defined set of APIs and libraries in return.
What if you need a library that is not in the runtime ? In that case, you can still bundle it with your application. You will be reponsible for the libraries that you bundle, but in return your users get exactly the version that you’ve tested your app with.
By decoupling the runtimes from the OS and by making them available in the same way as the applications, we make it possible to have applications that can run on different distributions, regardless of the library versions that are included in the OS. Of course, there is limits to what we can achieve: the applications and runtimes still have requirements (e.g. on Linux kernel features, or on session services that are expected to be present). But building an application that can run on Fedora, Ubuntu, Arch, RHEL and other modern distributions is entirely possible.
But what about security updates? To make this system work, we need to offer well-maintained runtimes that application authors can rely on, and provide updates for them just as we do now with distribution packages.
But that is not enough. We also need to isolate applications from the rest of the system (and your data!) at runtime, in order to limit the damage that they can do. Historically, the Linux desktop has been really bad about this: X lets any client snoop all input, and applications are generally free to read whatever they find in your home directory. Part of our desktop modularity story is to use container technologies and e.g. Wayland to confine (or sandbox) applications at runtime.
Modularity now!
We have been working towards this goal for quite a while, with the Wayland port and gnome-software as a non-package-centric installer.
What I’ve outlined above is more or less the architecture of the Flatpak system (formerly known as xdg-app). It is not 100% fleshed out yet, but enough pieces are in place now that you can try it out and e.g. install a nightly build of libreoffice that works on both Fedora and Ubuntu.
Head over to www.flatpak.org to learn more.
Please don’t call it modularity.
If this is one thing, it’s definitely not modular.
flatpacks are monoliths, not modular components.
Better described as a container format.
> flatpacks are monoliths, not modular components.
They are as modular as you write them to be
They are modular in the sense that you can install any Flatpak app on any distribution that has Flatpak. It doesn’t matter what version of which distribution that is or what other things you have installed. And that’s not possible with traditional package managers like rpm and dpkg.
Of course, the modules are bigger with Flatpak (whole apps instead of single packages), but they are much more independent from one another.
As I understand, current UX (http://flatpak.org/#users) is not the final one. Situation where user must enter bunch of console commands is not exactly user friendly.
How is it going to look like in the end? Now, user are used to getting an installer and double clicking it. Is Flatpak aiming at this kind of experience?
What about well known runtimes? Will I be able to define in my application manifest that it needs runtimes such and such available there and there without bundling them?
Right now, to run GNOME 3.20 from Flatpak, I had to first meddle with gpg keys and manually install gnome 3.20 runtime before installing an application.
Another thing. Is Flatpak going to cover use case for users that don’t have administrative rights on their machines (but have flatpak already installed) ?
gnome-software 3.21 has initial support for installing flatpaks, 3.22 will be much more complete. It is using the flatpak library, other software installers can do the same. flatpak supports installing into $HOME and installing system-wide (using a system helper), and I believe gnome-software will also offer this choice in 3.22.
This is good to hear. I’m really looking forward to it.
Maybe this info could be added to FAQ or some Roadmap page on flatpak.org because right now it gives bad idea of flatpak UX compared to e.g. AppImage which heavily emphasizes ease of use (click-and-run)
Oh, I haven’t noticed one paragraph that mentions it.
Anyhow, Roadmap would be nice 🙂
Didnt NixOS solve these issues years ago? I dont understand why that research is getting ignored?
Right now we can’t develop against GNOME SDK 3.20 when you depend in third party libraries, because by design SDK is read only, then we can’t install these libraries while you hack in your project.
Will be something in place in 3.22? Or at least included in the road map?
Developing against the SDK does not mean adding things to it. If a library is not in the runtime/SDK, you must include it as part of your applications build. See https://git.gnome.org/browse/gnome-apps-nightly/tree/org.gnome.Documents.json for an example of how that looks in practice.
This sounds like a good first step. Will we be able to declare runtime dependencies with a version wildcard? For example, my app could be declared dependent on GTK+ 3.18+ to show that it will work on anything between 3.18.1 until the current release. This would allow developers to not hardcode dependencies, reduce runtime dependencies and “thin out the dependency graph” as you mentioned.
The idea behind runtimes is that they need to be stable, and long-term supported. We want an application that has been built against org.gnome.Platform/3.18 to keep using that runtime (which will receive updates for 3.18.1, 3.18.2, etc). This is important to avoid problems like we’ve seen in 3.20 when GTK+’s theming infrastructure got an overhaul, and many applications needed fixes. If they had been shipped as flatpaks built against the 3.18 runtime, they would have kept using that until the app author had all the fixes in place and is ready to switch the build over to the 3.20 runtime. One of the big advantages here is that you can have the 3.18 and 3.20 runtimes installed at the same time, and apps will use what they have been built against. That is a big difference to the current distribution model where library updates get forced down every applications throat, read or not.
Will runtimes be able to depend on each other?
I mean, it seems that it would make sense to have “gtk+” runtime for people wanting to write applications for gtk but withouth dependencies on gnome.
Gnome runtime could then depend on gtk runtime.
Have you thought about this kind of scenario?
If you only want to depend on “gtk” but not “gnome” (ignoring, for a second, that most of the functionality you call “gnome” is really “gtk”) then you can just depend on the Freedesktop.org runtime, and bundle your version of GTK+ with your application.
AFAIK this is not planned. When I asked about it I was told that Flatpak authors didn’t want to build yet another dependency system and it’s probably a good decision. OSTree can do some sort of deduplication if contents of runtimes overlap.
Why? What advantages would there be in having a gtk runtime in addition to the gnome runtime? (Apart from potentially less SSD/HDD usage, but if you’re using Flatpak, you’re probably not that concerned about that.)