In the previous parts (1,2,3,4) we created various applications and tested them locally. Now we will look into how this application can be distributed to your users.
First we need to talk a bit about OSTree. This is the core distribution and installation mechanism of xdg-app. OSTree is similar to git, but has been designed to handle trees of large binaries. Just like git it has the concept of repositories, commits and branches (refs) . Branches are names which points to a commit id, and an application stored in a repository is such a branch.
On your machine there is a local OSTree repository which has a bunch of “remotes” configured. When you install or update an application from one on these remotes, what happens is that we ostree pull the branch from the remote, then do a local checkout of the branch (with hard links back to the repository files, so this is cheap).
The OSTree repositories created by xdg-app build-export are of a type (called archive-z2) that are meant to be served by a dumb webserver. So, to distribute your application, all you have to do is copy the repository to a webserver and give the url to your users.
There are however some extra details you have to be aware off .
First of all, the format of the archive-z2 repositories is one file per file in the app. This means a pull operation will do a lot of http requests. Since new requests are slow, make sure you enable support for HTTP keep-alive in your webserver.
OSTree also supports something called static deltas. These are single files in the repo that contains all the data needed to go between two revisions (or from nothing to a revision). Creating such deltas will take a bit more space, but will make downloads much faster.
xdg-app has the build-update-repo command that is very useful to manage repositories. Starting at version 0.4.13 it supports --generate-static-deltas to generate the deltas. but it also has several other features that are useful. For instance, you can use:
xdg-app build-update-repo --title="Nice name" repo
To set a user readable name for the repository, which will be used by default in the UI when users see this repository.
It also lets you prune (--prune) unused objects and deltas from the repository, and even remove older revisions from your repository (--prune-depth) which is useful for things like automatic nightly build repositories.
Another thing build-update-repo does is appstream extraction. It scans all the branches in the repository and looks for an AppStream xml file. For instance, the dictionary app from part 3 contains such a file in files/share/app-info/xmls/org.gnome.Dictionary.xml.gz. All these files plus the icons they reference are collected and commited into a repo-wide appstream branch. xdg-app keeps a local copy of this branch for each remote, which you can manually update with:
xdg-app --user update --appstream nightly
This is used by graphical installation tools such as gnome-software, so having appstream data for your applications is important.
Note: xdg-app-builder automatically runs the appstream-compose command after the build. This will collect information from appdata files and desktop files and create the right xml.gz and icon files. This is where the xml in gnome-dictionary came from.
Another important part about app distribution is gpg signatures. By default OSTree refuses to pull anything from a remote repository that is not signed. This is only allowed if you use --no-gpg-verify when you add the remote (or you can change it with xdg-app remote-modify).
In OSTree, signatures are on each commit and on the summary file that lists all the branches in the repositories. These objects are created by the build-update-repo, and build-export commands, as well as indirectly by xdg-app-builder. So, when you call these, you should pass in the gpg keys to sign with, and optionally the gpg home directory to use. For example:
xdg-app build-export --gpg-sign=KEYID --gpg-homedir=/some/dir appdir repo
That concludes this tutorial series, and you should now know enough to package and distribute your own applications.
I loved you series it was quite enlightening how this all fits together and to get a nice overview of the big picture. What i personally missed was “how to make a runtime”. Not that you promised that explicitly but i was somehow expecting that after reading that in part 2:
”
We have now built a simple application with no dependencies. If there are any dependencies that are not in the runtime you need to build those too. This means more cycles of configure; make; make install; While this is not hard, it is a lot of manual repeated work. In the next part of this series we will see how this can be automated using the xdg-app-builder tool.
”
So for now, i cant really use xdg-app – because i cannot build my own runtime – for projects more complex than a hello world one.
Nevertheless keep up the great work, hope to see xdg-app on various systems in the near future 🙂
I don’t understand why you feel the need to make a runtime. That is equivalent to making your own distro, and is not something normal developers should do.
I realize that i somehow forget to ask for that now that i read your comment. That was a question i had but forget to write that down – silly me. Now you answered in advance if its sane to build your own runtime. So after thats clarified it seems that i just had to play that old waiting game for suitable runtimes. Sry for being slightly to enthusiastic.
Xdg-app is a bundling system. The major way to get software dependencies into your app is to bundle them, and xdg-app-builder makes this pretty easy.
The runtime/app split is done in order to split the maintainance load to different groups. Anyone maintaining a runtime needs to know about really low-level technical stuff and be ready for zero days in glibc or openssl on the weekend. This is not something app authors should have to be responsible for, thus the split.
If you use the freedesktop runtime for instance you get the basic Linux runtime plus mesa, xorg/Wayland, dbus, sdl2, pulseaudio, etc. What is it you need to be in the runtime above this that you can’t bundle?
Would it make sense to build stores like steam, gog and smartphone app stores on top of xdg-app in the future? I think these stores usually only have at most one runtime, so xdg-app should be strictly superior in that regard. Sandboxing and true distribution independence would be additional advantages. Or are there reasons not to do this?
Also, most stores want 30% of the cut. What is stopping us from having a store where 100% of the cut go to the developers? I mean, sure, the distributors have to pay for the servers and traffic, but on the other hand Fedora/Ubuntu/etc. are doing that for free already.
Making a store would be interesting, but anything related to money tends to be very complicated.
Just wanted to say thanks for the series. While I don’t have any setup for distributing it yet, Meld in master now has support for doing xdg-app builds and the whole process seems to… just work. Thanks!
I have no idea whether it’s on the roadmap, but having pygobject + introspection support in the default gnome runtime would be extremely convenient, at least for us.
Python 3 pygobject support should be there.
That’ll be it then. Meld is still Python 2, though hopefully that will change soon. Thanks.
Hi, thanks for the great introduction. Really looking forward to xdg-app.
I am the author of an Markdown editor and I rely on Pandoc to convert documents to other formats.
Naturally I would like to keep pandoc as a system dependency and not compile it (and also texlive) myself. However, as far as I know, pandoc does not offer a DBUS interface.
Am I out of luck? Or is it possible to use system dependencies?
wolfv: Well, texlive and stuff are *large* dependencies, that most apps will not needs. As such they don’t really match the goals of what is in a runtime, so bundling is the typical solution here.
I understand its somewhat of a pain to bundle it, but the same is true for almost every dependency, and we can’t put everything in the runtime.
Why don’t you allow to specify various dependencies in terms of xdg-apps instead of one monolithic runtime? I mean, sort of a beafed up package manager with sandboxing.
I mean, there are a lot of packages that don’t need gnome, but a lot of other common dependencies. It would be much nicer to package these dependencies independently and than simple state the dependence in the apps meta files.
So essentially what I’m trying to ask is: why one monolithic runtime vs. multiple finer grained dependencies?
PS: what about the performance with all this sandboxing?
PPS : I do like what you have done, though. Thanks for all the work, looking forward to using it.
Dependencies are a great way to construct a build of something. For instance you can use packages and their dependencies to create a runtime or an application. You can do this very well with xdg-app/flatpak.
However, I don’t think they are a good way to deliver the final bits to the user. You want to test and verify the exact bits that your users will be running, not have them combine various versions of the dependencies at runtime, updated in different order, running post-install scripts and whatnot. That is not a good way to deliver robust, well-tested code.
In general the performance loss from the sandboxing should be very marginal. The main thing that is slower is dbus requestes, since we’re doing userspace filtering on that
Well, I assumed that it is possible to run multiple versions of a software side-by-side. If so, this should hold for dependencies as well, which means, you simply specify the exact version of the dependencies in your app. From a robustness point of view, this should be equal to package everything into one with the specified version.
I think the current approach is great, for apps that are part of a larger ecosystem like Gnome, but not for more individual apps with finer dependencies without needlessly increasing the build effort and artifact size. I would argue that most people use these kind of apps because of their few deps and their small footprint.