Category: General

Welcome back to this multi-part tutorial in how to create xdg-app applications. In part 2 we built a an xdg-app for gnome-dictionary. However, if you look closely at the commands we used you will set not much what we did was specific to this application.

In fact, the only things related to gnome-dictionary are the:

• application id
• source url
• binary name
• necessary permissions

Why is this?

It turns out that most open source applications are built in very similar ways. One can even consider this an API for building modules. And if some module does not conform to this API, then it is easy to change the upstream to conform, or if upstream is not interested, apply a local patch.

The xdg-app-builder tool that ships with xdg-app is based on this idea. You describe your application, and the modules you want to build into it, and then xdg-app-builder takes care of calling the lower-level xdg-app build commands for you.

The equivalent of what we did in part 2 is this json:

{
  "app-id": "org.gnome.Dictionary",
  "runtime": "org.gnome.Platform",
  "runtime-version": "3.20",
  "sdk": "org.gnome.Sdk",
  "command": "gnome-dictionary",
  "finish-args": [ 
     "--socket=x11", 
     "--share=network"  
  ],
  "modules": [
    {
      "name": "gnome-dictionary",
      "sources": [
        {
          "type": "archive",
          "url": "https://download.gnome.org/sources/gnome-dictionary/3.20/gnome-dictionary-3.20.0.tar.xz",
          "sha256": "efb36377d46eff9291d3b8fec37baab2355f9dc8bc7edb791b6a625574716121"
        }
      ]
    }
  ]
}

If you put this in a file called org.gnome.Dictionary.json you can build and export the app using:

$ xdg-app-builder --repo=repo appdir3 org.gnome.Dictionary.json

And to test this you use:

$ xdg-app --user update org.gnome.Dictionary
$ xdg-app run org.gnome.Dictionary

So the above commands downloaded the files, initialized the build dir, built the modules, finished the app and exported it to the repo. Just like we did before.

However, even for this simple file it did a lot of other nice things too.

• It verified the sha256 checksum of the downloaded tarball
• It built all sources in a fixed location (in /run/build) to ensure more repeatable builds.
• It ran the builds without access to any part of the host filesystem, other than the directory with the extracted sources. This means less chance of the build machine details affecting the build.
• It automatically extracted all the debug information from the installed binaries into separate files, and these were commited to a separately installable runtime called org.gnome.Dictionary.Debug.
• Translations were also extracted to separately installable runtimes called org.gnome.Dictionary.Locale.$lang
• It cached each stage of the build, so that if you need to rebuild the app only the modules that have changed will be rebuilt.

Other than this xdg-app-builder has a lot of useful features. The obvious one is that you can build multiple modules into the application, and you can have multiple sources for each module. Sources can be of several types. Currently it supports: archive (.tar, .zip), git, bzr, patch files, and just running shell commands.

There is also a cleanup phase that happens after the build. This phase lets you remove things that was added during the build, that will not be needed during runtime. For instance, you can remove headers, development docs and similar things here. xdg-app-builder supports two properties for this. First a list of filename patterns, and secondly a list of commands to run during the cleanup phase.

"cleanup": [ "/include", "/bin/foo-*", "*.a" ]
"cleanup-commands": [ "sed s/foo/bar/ /bin/app.sh" ]

Additionally, the cleanup property can be set on a per-module basis, and will then only match filenames that were created by that particular module.

Another special feature is the rename-icon, rename-desktop-file and rename-appdata properties which lets you rename these kinds of files to match the application id. You have to do this because upstream files don’t use the application id in the name, which is not allowed by xdg-app during export.

The nightly build of the gnome applications all are built using xdg-app-builder, and you can get a lot of example json files there, for instance a complete version of gnome-dictionary built from git.  I also have a few more examples here.

For detailed documentations on all xdg-app-builder properties. See the man page for xdg-app builder.

That is it for today. Next time we will take a look at some details about the xdg-app sandbox.

Welcome back to this multi-part tutorial in how to create xdg-app applications. In part 1 we installed everything we needed and manually created our first application. In this part we will build a more complex application, using the basic xdg-app tools.

I chose to build the gnome dictionary application, because its a small app with no dependencies. To build it we use the SDK (software development kit) that matches the runtime we used in the previous part. The SDK contains the development toolchain and the headers needed to build applications using the libraries in the runtime.

First we need to install the SDK. This is rather large, so it may take a while:

xdg-app --user install gnome org.gnome.Sdk 3.20

We also want the sources to the application we want to build:

wget https://download.gnome.org/sources/gnome-dictionary/3.20/gnome-dictionary-3.20.0.tar.xz

Just like last time we start by creating the application directory where the application will be installed. However, this time we use the build-init command to create it:

xdg-app build-init appdir2 org.gnome.Dictionary org.gnome.Sdk org.gnome.Platform 3.20

This creates a directory called “appdir2” with a metadata file and the right directory structure. The only difference from last time is that the metadata file specifies a sdk as well as a runtime. A sdk is really just a runtime, but it is used during the build phase, rather than when the user runs the app.

Now that we have specified a sdk we can use the xdg-app build command. This is very similar to the run command, except it operates on an application build directory rather than an installed application, and it uses the sdk instead of the regular runtime. It also gives the sandbox access to all your files. For example, you can run:

$ xdg-app build appdir2 touch /app/some_file
$ xdg-app build appdir2 ls -l /app
total 0
-rw-r--r-- 1 alex wheel 0 Feb 19 16:03 some_file

This will have created a file “appdir2/files/some_file” in your tutorial directory.

Using this command you can build the application like you normally would, except in a sandbox. Since it is in a sandbox it will automatically use the compiler and other tools from the sdk. gnome-dictionary builds fine with a traditional configure; make; make install; incantation, so we only have to add a xdg-app prefix to these:

tar xvf gnome-dictionary-3.20.0.tar.xz
cd gnome-dictionary-3.20.0/
xdg-app build ../appdir2 ./configure --prefix=/app
xdg-app build ../appdir2 make
xdg-app build ../appdir2 make install
cd ..

We also need to give the app access to X11 and the network, and specify the command that is used to start the app. This is done with the build-finish operation:

xdg-app build-finish appdir2 --socket=x11 --share=network --command=gnome-dictionary

This adds some extra info to the metadata file, and creates the exports directory, which we’ll come back to later. You can install and try the app by running

xdg-app build-export repo appdir2
xdg-app --user install tutorial-repo org.gnome.Dictionary
xdg-app run org.gnome.Dictionary

You will see some warnings because we didn’t grant the app access to the dconf database, but other than that everything works.

On interesting new thing here is the exports mentioned above. If you look into the appdir2/export/ directory you will find this structure:

appdir2/export
└── share
    ├── applications
    │   └── org.gnome.Dictionary.desktop
    └── dbus-1
        └── services
            └── org.gnome.Dictionary.service

And when the app is installed, these files will be exported into ~/.local/share/xdg-app/exports/. The xdg-app package then sets the XDG_DATA_DIRS environment variable to point to this directory, which means your desktop environment will look in it. This sounds a bit complicated, but what it means is that once the app is installed it will automatically appear among the normal applications in your desktop environment.

There is one limit on what gets exported though. All the filesnames (not directory names) must have the application id as a prefix. This works in the above setup because the desktop file is org.gnome.Dictionary.desktop, which is what we used for the id. This limitation guarantees that applications cannot cause conflicts, and that they can’t override any system installed applications.

We have now built a simple application with no dependencies. If there are any dependencies that are not in the runtime you need to build those too. This means more cycles of configure; make; make install; While this is not hard, it is a lot of manual repeated work. In the next part of this series we will see how this can be automated using the xdg-app-builder tool.

Welcome to part 1 in this multi-part tutorial in how to create xdg-app applications.

First of all you need to install xdg-app. Fedora 23 ships with xdg-app packages, but some distributions do not. I have made this page with packages that you can use.

This tutorial is using xdg-app 0.4.12, which is the latest version. Older versions should work too, but minor things may need to be changed in the commands, and not all features may be supported.

A fundamental concept in xdg-app is the runtime/application split. Every app depends on a runtime, which supplies the core libraries that the app relies on. Runtimes are typically shared by many applications, but a user can have multiple runtimes installed at the same time.

For this tutorial we will be using the gnome nightly build runtime. It is available in a xdg-app repostitory on sdk.gnome.org. If you tried the gnome nightly builds from my previous blog entry you will have these already, but otherwise you can get it by doing:

$ wget https://sdk.gnome.org/keys/gnome-sdk.gpg
$ xdg-app --user remote-add --gpg-key=gnome-sdk.gpg gnome http://sdk.gnome.org/repo
$ xdg-app --user install gnome org.gnome.Platform 3.20

Once it is installed you can also update it by doing:

xdg-app --user update org.gnome.Platform 3.20

Now we have all that we need to create a simple application. Lets start by writing an application. Put this in a file called “hello.sh”:

#!/bin/sh
echo "Hello world, from a sandbox"

Then we need some information about the application. In xdg-app this is in specified in a key-value file called “metadata”. For a simple app like this it doesn’t contain much, so we can create it manually:

[Application]
name=org.test.Hello
runtime=org.gnome.Platform/x86_64/3.20
command=hello.sh

This specifies the application identifier (org.test.Hello) as well as the runtime the application uses and the command to start the app with.

An xdg-app really doesn’t require more than this, although the application must have a special layout. Using these sources we can create an application directory “appdir” like this:

mkdir appdir
mkdir appdir/files
mkdir appdir/files/bin
mkdir appdir/export
cp metadata appdir/
cp hello.sh appdir/files/bin/
chmod a+x appdir/files/bin/hello.sh

In order for xdg-app to be able to install this application we need to put it in an  repository. This is done with the build-export command:

xdg-app build-export repo appdir

This will initialize a local repository in the directory “repo”  and export the app to it. We can test it using:

xdg-app --user remote-add --no-gpg-verify tutorial-repo repo
xdg-app --user install tutorial-repo org.test.Hello
xdg-app run org.test.Hello

This will print “Hello world, from a sandbox”, which means we just created, packaged, installed and ran our first xdg-app application!

If you want to examine the sandbox a bit you can use the –command=sh argument to xdg-app run, which will give you a shell inside the application sandbox where you can explore the sandbox. For example:

[alex@localhost ~]$ xdg-app run --command=sh org.test.Hello
sh-4.3$ ls -lR /app
/app:
total 0
drwxr-xr-x 1 alex wheel 16 jan 1 1970 bin
/app/bin:
total 4
-rwxr-xr-x 2 alex wheel 45 jan 1 1970 hello.sh
sh-4.3$ echo $PATH
/app/bin:/usr/bin
sh-4.3$ hello.sh
Hello world, from a sandbox
sh-4.3$ ls -la ~/
total 0
drwxr-xr-x 3 alex wheel 60 feb 19 16:11 .
drwxr-xr-x 3 alex wheel 60 feb 19 16:11 ..
drwxr-xr-x 3 alex wheel 60 feb 19 16:11 .var
sh-4.3$ ls ~/.var/app/org.test.Hello/
cache  config  data

Here we see that all the files from the application appears in /app, and PATH points to /app/bin which means you can easily run it. We also see that by default the application has no access to any files from the users home directory (it’s empty) other than the ~/.var/app/org.test.Hello directory where the app can store its own data. You can also explore /usr, which comes from the runtime.

This is a very manual approach to creating a xdg-app, in the next part of this tutorial we will look at how to use the “xdg-app build” command and an SDK to build applications.

After clicking the button:

I’m here at the gnome Developer Experience Hackfest in Brussels working on xdg-app. Just before I left I created a runtime for Gnome based on whatever is in git master. Now we’ve started to create app bundles for the gnome applications.

I’ve added builds of evince, gedit, gnome-builder, and maps. Cosimo has added Weather, and clocks and Alberto added gnome-calculator. The build manifests for these are in my github repo, and I have set up an automatic build of these (and the SDK).

Unfortunately the build machine is way underpowered, so its not yet useful for public consumption. I’m working on getting this to build on the gnome build machines, which means people can start testing the latest builds of gnome apps on any distros.

I’ve also been working with Simon to fix various issues that he’s seeing while packaging xdg-app for debian.

On to building more apps!

Thanks to the gnome foundation for sponsoring this trip, and arranging the hackfest.

Yesterday I released xdg-app 0.4.6 and I wanted to take some time to talk about what is new in this version what is happening around xdg-app.

libxdg-app and gnome-software integration

In the release, but disabled by default, is a new library called “libxdg-app”. It is intended for applications that want to present a user interface for managing xdg-app applications. We’re working on integrating this with gnome-software so that we can have graphical installation and updating of applications. This is work in progress, and the APIs are not yet stable, but it is very important progress that we will continue working on in the near future.

New xdg-app-builder tool

The basics of how to bundle and application with xdg-app is very simple. You initialize an application directory with build-init. For example:

$ xdg-app build-init appdir 
          org.example.ExampleApp 
          org.gnome.Sdk org.gnome.Platform 3.18

This gives you an place where you can both run the build, and store the application being build. Typically you then go to your source directory and run something like:

$ xdg-app build appdir ./configure --prefix=/app
$ xdg-app build appdir make
$ xdg-app build appdir make install

At this point the application is mostly done, but you need to run build-finish in order to export things like desktop files and icons as well as configure some application metadata and permissions, and then export the directory to an ostree repository that your users can install it from:

$ xdg-app build-finish appdir
     --command=run-example --socket=x11
     --share=network --filesystem=host
$ xdg-app build-export appdir /path/to/repo

This is pretty easy, as long as all the tools you need to build your app are in the sdk, and all the dependencies the app needs are in the runtime. However, most apps need a few extra dependencies, which was a large pain point for  people experimenting with xdg-app.

I decided to write a tool that automates this, and thus xdg-app-builder was born. It builds on experience from the Gnome continuous integration system and the nightly xdg-app build work that I did a while ago. Its based on the build-api proposal from Colin Walters, and the idea is to push as much build-knowledge upstream as possible, so that all you need to do is list your dependencies.

Here is an example json manifest that describes the above steps, plus adds a dependency:

{
  "app-id": "org.example.ExampleApp",
  "version": "master",
  "runtime": "org.gnome.Platform",
  "runtime-version": "3.18",
  "sdk": "org.gnome.Sdk",
  "command": "run-example",
  "finish-args": ["--socket=x11", 
                  "--share=network", 
                  "--filesystem=host" ],
  "build-options" : {
    "cflags": "-O2 -g",
    "env": {
        "V": "1"
    }
  },
  "cleanup": ["/include", "*.a"],
  "modules": [
    {
      "name": "some-dependency",
      "config-opts": [ "--disable-something" ],
      "cleanup": [ "/bin" ],
      "sources": [
        {
          "type": "archive",
          "url": "http://someting.org/somethinbg-1.0.tar.xz",
          "sha256": "93cc067b23c4ef7421380d3e8bd7c940b2027668446750787d7c1cb42720248e"
         }
       ]
    },
    {
      "name": "example-app",
      "sources": [
        {
          "type": "git",
          "url": "git://git.gnome.org/gimp"
        }
      ]
    }
  ]
}

In addition to just building things this will also automatically download tarballs and pull git/bzr repos and clean up and strip things after install. It even has a caching system so that any module that did not change (in the manifest, or in the git repos) will have the results taken from the cache on consecutive builds, rather than rebuilding.

Some people have started using this, including the pitivi and glom developers, and I’ve converted the existing nightly builds of gimp and inkscape to use this instead of the custom scripts that was used before. If you’re interested in playing with xdg-app-builder those links should give you some examples to work from. There is also pretty complete docs in the manpages for xdg-app-builder.

Updated nightly builds

As I mentioned above the nightly builds were converted to xdg-app-builder, but I have also extended the set of builds with Darktable, MyPaint and Scribus, in addition to the old Gimp and Inkscape builds. The scribus build have some issues which I don’t understand (help needed), but the others seem to work well.

If you’re interested in using these, take a look at https://wiki.gnome.org/Projects/SandboxedApps/NightlyBuilds which has instructions on how to get builds of xdg-app for your distro and how to use it to test the nightly builds.

Updated runtime and sdk

Since more people have started testing the Gnome runtimes I’ve fixed quite a few issues that were found in them, as well as added some new tools to the sdk. If you installed the old one, make sure to update it.

Upcoming work

The basic functionality of xdg-app is pretty much there, at least for non-sandboxed applications. The main focus of the work right now is to finish the integration with gnome-software. But after that I will return to work on sandboxing, finishing the work on the file chooser portal and the other APIs required to run apps in a sandboxed fashion.

Recently I have been working on support for a filechooser portal for sandboxed applications (in xdg-app). The way these work is that the application triggers a file chooser, then the actual user interaction happens outside the sandbox, and the application only recieves the file data after the user finished the interaction.

Ideally something like this would be completely hidden by the toolkit, and the application would just use the regular file chooser APIs. However, the Gtk+ filechooser APIs expose too much details about the file chooser dialog, which means it has to be a regular in-process widget. Unfortunately this means we can’t replace it by an out-of-process dialog.

What we need is a Gtk+ API for the file chooser that hides the details of how the dialog works. At this point I realized that this is something that has been requested a lot in a different context. Such an API would allow us to plug in platform-native file chooser dialogs.

So, I got to work, and today I landed support for native Windows file choosers in Gtk:

We will also look at implementing an OSX version of this to ensure that the APIs work for the common cases.

Now that we have this people can start porting their applications (which is quite trivial in the common case, as the APIs is very similar to the old APIs). And once the apps are ported they will automatically get support for the filechooser portal to make them work in a sandboxed environment.

So, anyone maintaining a Gtk 3 application that want to work better on other platforms, or when sandboxed should take a look at the new GtkFileChooserNative API. Right now it only exists in git, but once there is a new release it should show up in the API docs.

When reporting a bug in some software it is often common that the developer asks you to check if the bug is fixed in the latest development version. This makes a lot of sense from the perspective of a developer that gets a lot of bug reports. However, unless you are very experienced with building software this is prohibitively hard.

This is an area where xdg-app shines, because it allows you to create binary builds of desktop applications that work on any distribution. In order to demonstrate this I set up an automated build system that builds Gimp and Inkscape from the development branch every day and produces a new binary that you can easily install and run:

To make it easy to use I also created packages of xdg-app for some common distributions.

For more information about how to use these builds, see the nightly builds page.

One of the core ideas of xdg-app is that users should be running the same build of everything that the developers tested on. Not only does this mean that you can trust the testing that went into the app, but it also means that an app can run on multiple distributions, and on different version of the same distribution.

However, an application does not have bundle everything. Instead the app specifies a dependency on a runtime, which contains the base system libraries. I like to compare this to dynamic libraries, where xdg-app is “dynamically linked” to its runtime, whereas container systems (like docker) are “statically linked” (by shipping a complete runtime in each app).

This may seem weird and contrary to the first paragraph, but it turns out that this is pretty much a requirement. We want third parties to be able to produce a binary that will keep running “forever”, but the base system may need fixes or support for new hardware. We can’t expect every vendor to rebuild every application (for example some old game) each time something needs fixing on the lower levels. So, therefore we allow updates to the runtime separately from the app (although any update must be compatible).

An app can only depend on one runtime, and everything not in the runtime must be bundled with the application. There is (by design) no way to depend on multiple runtimes, or have runtimes depend on each other. However, there is something called runtime extensions. Extensions are a way to split off and make optional parts of the runtime and recombine them at runtime.

For instance, I’m working on a runtime called org.freedesktop.Platform that has the basic freedesktop libraries (X11, Mesa, DBus, etc). It has this snippet in the configuration:

[Extension org.freedesktop.Platform.Timezones]
directory=share/zoneinfo

This means that whenever another runtime called org.freedesktop.Platform.Timezones is installed its contents will replace the directory share/zoneinfo in the runtime. This is very useful as it allows the timezone info (which changes frequently) to be updated separately from the runtime.

It also has this:

[Extension org.freedesktop.Platform.Locale]
directory=share/runtime/locale
subdirectories=true

This means that if a runtime like org.freedesktop.Platform.Locale.sv is installed, it will replace the contents of share/runtime/locale/sv in the runtime. During the build all the locale data and translations are separated out into per-language runtimes which can be installed separately.

And finally, it has:

[Extension org.freedesktop.Platform.GL]
directory=lib/GL

There is no (official) runtime with this name, but if one is installed it will appear in lib/GL, and the main runtime has been programmed to look into this directory for libGL.

The idea here is that if your system uses an OpenGL driver that does not ship with the regular runtime (i.e. Mesa), or needs a more recent version of it, then you can create your own runtime with this name and get your drivers into the runtime.

For a long time this has been a theoretical solution, but recently I acquired an NVidia card in order to test this. The result is this script, which takes an upstream nvidia driver release and converts it into a runtime that matches the (soon to be released) 1.2 version of the Freedesktop runtime.

To verify that it works I created an xdg-app bundle for the Unreal Editor. Here are some screenshots of a sandboxed version of unreal to show this working:

While runtimes can’t have dependencies on other runtimes, they can be build from the same base, and thus be compatible. For instance, the official Gnome runtime takes the Freedesktop runtime and adds the Gnome modules to it. Since we have the same ABI  we can reuse the same extensions. The two runtimes have different versions (Gnome is 3.18, Freedesktop is 1.2), so we have to specify the version (which is otherwise infered by the runtime version). Here is how the Gnome runtime config looks:

[Extension org.freedesktop.Platform.GL]
version=1.2
directory=lib/GL

I hope to have stable builds out of the Freedesktop 1.2 and Gnome 3.18 runtimes shortly, so that other people can play with them. Unfortunately I’m not allowed to distribute the unreal editor app.