For anyone following the development of xdg-app, all development have now moved to freedesktop.org. Here is where things are happening now:
An important aspect of xdg-app is application sandboxing, which will require application changes to use sandbox-specific APIs. However, xdg-app is also a good way to deploy and run non-sandboxed (or partially sandboxed) regular applications.
A very interesting usecase for this is to have an image-based operating system, for instance a Workstation spin of Fedora Atomic. Such a system would have a basic workstation installation with a read-only /usr, and atomic updates/rollback. However, installing an application is painful, and customizing yor install in that way undoes many of the advantages of an image-based OS.
With xdg-app you can install apps into /var (or $HOME) and have them fully integrate with the system, while still being isolated from changes to the host. This makes for a great combination, just like atomic + docker is a good combination for the server space.
I’ve spent some time recently making a prototype runtime based on the Fedora packages, as reported on the desktop list. This is kind of interesting as it lets you test applications from rawhide on fedora 21 or 22. Just install xdg-app from fedora-updates and then install the runtime:
$ xdg-app add-remote --no-gpg-verify --user fedora http://fedorapeople.org/~alexl/repo/ $ xdg-app install-runtime --user fedora org.fedoraproject.Platform 23
And then you can try gedit 3.17.0:
$ xdg-app install-app --user fedora org.gnome.gedit $ xdg-app run org.gnome.gedit
Or evince 3.17.2:
$ xdg-app install-app --user fedora org.gnome.evince $ xdg-app run org.gnome.evince
Once installed you can also just start them from the desktop environment as usual. They should be there like any regular application as the desktop files and icons are exported to the host.
As people who have followed the work on sandboxed applications know, we have promised a developer preview for GNOME 3.16. Well, 3.16 has now been released, so the time is now!
I spent last week setting up an build system on the GNOME infrastructure, and the output of this is finally available at:
This repository contains the gnome 3.16 runtimes, org.gnome.Platform, as well as a smaller one that is useful for less integrated apps (like games) called org.freedesktop.Platform. It also has corresponding develoment runtimes (org.gnome.Sdk and org.freedesktop.Sdk) that you can use to create applications for the platforms.
This is a developer preview, so consider these builds weakly supported. This means I will try to keep them somewhat updated if there are major issues and that I will keep them API and ABI stable. I will probably also pick up at least some 3.16.x minor releases as they are released.
I also did the first official release of xdg-app. For easy testing this is available for Fedora 21 and 22 as a copr repo.
Using the repo above makes it really easy to test this. Just install the xdg-app package from copr, log out+in (needed update the environment for the session), then follow these instructions (as a regular user):
$ xdg-app add-remote --user gnome-sdk http://sdk.gnome.org/repo/ $ xdg-app install-runtime --user gnome-sdk org.gnome.Platform 3.16 $ xdg-app install-runtime --user gnome-sdk org.freedesktop.Platform 1.0
$ xdg-app install-runtime --user gnome-sdk org.gnome.Platform.Locale.se 3.16 $ xdg-app install-runtime --user gnome-sdk org.freedesktop.Platform.Locale.se 1.0
$ xdg-app add-remote --user --no-gpg-verify test-apps https://people.gnome.org/~alexl/test-apps/repo/ $ xdg-app install-app --user test-apps org.gnome.gedit $ xdg-app install-app --user test-apps org.freedesktop.glxgears
$ xdg-app run org.gnome.gedit $ xdg-app run org.freedesktop.glxgears
$ xdg-app install-runtime --user gnome-sdk org.gnome.Sdk 3.16 $ xdg-app install-app --user test-apps org.gnome.Builder
All the above install the apps into your home-directory (in ~/.local/share/xdg-app) . You can also run the commands as root and skip the –user arguments to do system-wide application installs.
With the basics now laid down to run current applications in a minimally isolated environment the next step is to work on the sandboxing aspects more. This will require lots of work, both in the system side (things like kdbus), the desktop (add sandbox aware APIs, make pulseaudio protect clients from each other, etc) and in modifying applications.
If you’re interested in this, you can follow the work on the wiki.
If you download the SDKs you have enough tooling to build your own applications. There are some documentations on how to do this here.
I also created a git repository with the scripts I used to build the test applications above. It uses the gnome-sdk-bundles repostory which has some tooling and specfiles to easily bundle dependencies with the application.
If you ever want to build the SDK yourself, it is available at:
https://git.gnome.org/browse/gnome-sdk-images
This repository contains the desktop specific parts of the SDK, which is layered on a core Yocto layer. When you build the SDK this will be automatically checked out and built from:
https://git.gnome.org/browse/freedesktop-sdk-base
However, if you don’t want to build all of this you can download the pre-build images from http://sdk.gnome.org/images/x86_64/ and put them in the freedesktop-sdk-base/images/x86_64 subdirectory of gnome-sdk-images. This can save you a lot of time and space.
Its not a secret that I’ve been working on sandboxed desktop applications recently. In fact, I recently gave a talk at devconf.cz about it. However, up until now I’ve mainly been focusing on the bundling and deployment aspects of the problem. I’ve been running applications in their own environment, but having pretty open access to the system.
Now that the basics are working it’s time to start looking at how to create a real sandbox. This is going to require a lot of changes to the Linux stack. For instance, we have to use Wayland instead of X11, because X11 is impossible to secure. We also need to use kdbus to allow desktop integration that is properly filtered at the kernel level.
Recently Wayland has made some pretty big strides though, and we now have working Wayland sessions in Fedora 21. This means we can start testing real sandboxing for simple applications. To get something running I chose to focus on a game, because they require very little interaction with the system. Here is a video I made of Neverball, running in a minimal sandbox:
In this example we’re running a regular build of neverball in an environment which:
Yet the application is still simple to install and integrates nicely with the desktop. If you want to test it yourself, just follow the instructions on the project page and install org.neverball.Neverball.
Of course, there are still a lot to do here. For instance, PulseAudio doesn’t protect clients from each other, and for more complex applications we need to add new APIs to safely grant access to things like user files and devices. The sandbox details page has a more detailed list of what has to be done.
The road is long, but at least we have now started our journey!
I’m now back from a week in Cambridge at the developer experience hackfest. This was a great event, it was a lot of fun to meet people again, and we got a lot of things done. I spent a lot of time talking to people about things related to xdg-app and sandboxed applications, both spreading information and actually implementing features.
I spent some time with Emmanuele, Ryan and Lars working on glib stuff, which resulted in the G_DECLARE_*_TYPE macros finally being merged. Additionally I reviewed the new list model abstraction which I hope we can land soon, and Ryan and I worked out a new fancy __attribute__(cleanup) approach that we hope to merge into glib soon.
We also worked a bit on Gtk+ OpenGL support. Based on feedback from early users we’re doing some changes in how GL contexts are created to allow you to configure them in more detail. We also decided that we want to completely drop support for legacy OpenGL contexts, as these had issues cooperating with Core 3.2 contexts, and because we don’t live in the 90s anymore. Carlos was working on converting GtkPopover to use (override redirect) toplevels on X11, and I gave him moral support and generally hated on ancient crappy X11 behaviour.
Props to Collabora and Philip for arranging a great event!
Over a few releases now, and culminating in Gtk+ 3.10 we have restructured the way drawing works internally. This hasn’t really been written about a lot, and I still see a lot of people unaware of these changes, so I thought I’d do a writeup of the brave new world.
All Gtk+ applications are mainloop driven, which means that most of the time the app is idle inside a loop that just waits for something to happen and then calls out to the right place when it does. On top of this Gtk+ has a frame clock that gives a “pulse” to the application. This clock beats at a steady rate, which is tied to the framerate of the output (this is synced to the monitor via the window manager/compositor). The clock has several phases:
The phases happens in this order and we will always run each phase through before going back to the start. The Input events phase is a long stretch of time between each redraw where we get input events from the user and other events (like e.g. network i/o). Some events, like mouse motion are compressed so that we only get a single mouse motion event per clock cycle[1].
Once the event phase is over we pause all external events and run the redraw loop. First is the update phase, where all animations are run to calculate the new state based on the estimated time the next frame will be visible (available via the frame clock). This often involves geometry changes which drives the next phase, Layout. If there are any changes in widget size requirements we calculate a new layout for the widget hierarchy (i.e. we assign sizes and positions). Then we go to the Paint phase where we redraw the regions of the window that needs redrawing.
If nothing requires the update/layout/paint phases we will stay in the event phase forever, as we don’t want to redraw if nothing changes. Each phase can request further processing in the following phases (e.g. the update phase will cause there to be layout work, and layout changes cause repaints).
There are multiple ways to drive the clock, at the lowest level you can request a particular phase by gdk_frame_clock_request_phase() which will schedule a clock beat as needed so that it eventually reaches the requested phase. However in practice most things happen at higher levels:
There are also a lot of implicit triggers of these from the CSS layer (which does animations, resizes and repaints as needed).
During the Paint phase we will send a single expose event to the toplevel window[2]. The event handler will create a cairo context for the window and emit a GtkWidget::draw() signal on it, which will propagate down the entire widget hierarchy in back-to-front order, using the clipping and transform of the cairo context. This lets each widget draw its content at the right place and time, correctly handling things like partial transparencies and overlapping widgets.
There are some notable differences here compared to how it used to work:
This skips some of the details, but with this overview you should have a better idea that happens when your code is called, and be in a better position to do further research if necessary.
[1] If you need more mouse event precision you need to look at the mouse device history.
[2] Actually each native subwindow will get one too, but that is not very common these days.
Connoisseurs of this blog know that I have an interest in application deployment systems, having created three different application bunding system (1,2, 3). These were all experiments in the area of desktop applications, but recently there has been some interesting motions in related areas, namely Docker.
Docker is a server application/container deployment system, which nicely sidesteps a lot of the complexity with desktop apps (not having to integrate deeply with the desktop) which makes it a lot easier to deploy. Additionally, docker is more than a deployment system, it also has some interesting ideas about how to create and distribute applications.
Every docker container is a copy-on-write clone of a specific parent Image, which means instantiation of docker containers is very fast and cheap. It also gives some very interesting properties because you can track the changes in a container (compared to t its parent image) and “commit” this to a new image. This creates a git-like hierarchy of images where every commit is a filesystem layer that applies to a previous layer, up to some base image. The git-like workflow is really nice to work with when creating images, and the final result is very easy to share and deploy, and at the same time automatically shares as much as possible with common base images.
Unfortunately Docker relies on AUFS, a union filesystem that is not in the upstream kernel, nor is it likely to ever be there. Also, while AUFS is in the current Ubuntu kernel it is deprecated there and will eventually be removed. This means Docker doesn’t run on Fedora which has a primarily-upstream approach to packaging.
So, the last month or so I’ve been working on making Docker work in Fedora (and thus eventually in RHEL, which is the nr 1 requested Docker feature). Of course, this work will benefit other distributions that don’t have AUFS too.
I started looking at possible replacements for the copy-on-write support, and there are a few possibilities availible:
Overlayfs is a different union filesystem implementation than AUFS, and the one that seems most likely to land upstream. But that is happening slowly, if at all. Long-term I think this is the best option, but right now it is out of the question.
Btrfs has copy-on-write both using filesystem snapshots and one a file basis using reflink. However, btrfs is not currently used much in production as its not considered stable enough. It would also be a very heavy dependency for Docker, as may users would have to reformat their disks to use it.
Lvm snapshots are useful for doing e.g. backup of a snapshot, but regress badly in performance when you start having many snapshots of the same device.
This leaves us with only lvm thin provisioning. This is a fairly recent, but relatively stable technology that allows you to create copy-on-write block devices that are “thinly” provisioned, meaning they don’t use real space until the device is in use. This is not ideal for Docker as it really wants copy-on-write at the file level, but with some work it is possible to work around this.
Rather than interacting with lvm which is a very generic volume manager I chose to use the lower level device-mapper kernel APIs directly (via libdevmapper). This allows us greater ease of access to the devices programmatically, as well as avoiding confusion with possible system use of LVM. Also it avoids some LVM performance issues with very many devices.
So, we set up a single large block device on which we create a device-mapper “thinp” pool. On this we then creates a single “base” block device formated with ext4. Every image and container are then created as snapshot (in multiple steps) from this base device. So, say you’re starting a container based on an image “apache” which itself is based on a “fedora” image, we would:
And of course, these devices will be reused (with corresponding steps skipped) as needed by other images/containers.
The devicemapper pool need to be set up on a large block device that fits all the images and containers that you will be used which would be painful for most people. Docker handles this by automatically creating the a large sparse file, using it as a loopback device for the devicemapper work. Additionally we ensure that DISCARD support is enabled in the filesystem so that any files removed in the conttainer filters down to the loopback file making it sparse again.
This means that there is no need for setup, and space for images and containers will only be used as needed. Of course, there are still issues, like the max size of the loopback mount (100G by default, but this should be easy to grow) and the max size of the base extt4 image (10G by default, resizing is harder after initial construction, but should be possible).
We’re currently in the process of landing this in Docker, and hope to have a 0.7 release out based on my device-mapper work pretty soon. Then I will continue working on making docker a first-class citizen on Fedora.
For some time I’ve been working on HiDPI support for Gnome, in order to support all the new laptops with very high resolution displays. This work is now at a stage where I can start showing it off and adventurous users might even want to test it out.
The support happens on many layers, like:
But most of the work has been in Gtk+. There is a branch called wip/window-scales that contains this code. It has:
Here is how it looks right now: (click for full version)
This is work in progress, but if you want to try it out, here is the code you need to use:
You can then enable the scaling either by setting GDK_SCALE=2 in the environment, or by using gsettings (only works under gnome):
gsettings set org.gnome.settings-daemon.plugins.xsettings overrides
"{ 'Gdk/WindowScalingFactor':<2>, 'Gdk/UnscaledDPI':<92160> }"
There is still a lot of work to do, but I hope to have this mostly done for my talk at Guadec. Hopefully I’ll see you there!
And last, but not least, many thanks to Brion Vibber who generously donated a Chromebook Pixel to me so that I could work on this stuff.
Since my last blog entry I’ve worked a bit more on broadway, and the openshift-broadway cartridge for openshift.
New in this version:
Here is a video of this:
[youtube width=”512″ height=”384″]http://www.youtube.com/watch?v=a8__4mF4-8g[/youtube]
Today is the third day of the Gnome Developer Experience hackfest. The first day I was in the platform group where we looked at the core gtk+ platform and whats missing from it. We ended up with a pretty large list of items, but we picked a few of them and started on a few of them. More details in cosimocs blog post.
Yesterday we finally had all the people interested in the application deployment and sandboxing story here, so we started looking at that. I’ve historically had a lot of interest in this area with previous experiments like glick, glick2, and bundler, so this is my main interest this hackfest. We have some initial plans for how to approach this.
There are two fundamental, but interrelated problems in this area. One is the deployment of the application (how to create an application, bundle it, “install” it, set up its execution environment and start it) and the second is sandboxing (protecting the user session against the app).
For deplyoment we’re considering a bundling model where the app ships the binary and some subset of the libraries it needs, plus a manifest that describes what the kind of system ABIs that the app requires. These are very course grained, unlike the traditional per-library dependencies, and would be on the level of i.e.
There is a scale here where apps (like games) can chose to bundle more dependencies, and expect more stability over time, but it will not be very integrated, while on the other hand more integrated desktop apps which will be tied to specific versions of some desktop platform.
What is important here is that once an app specifies a particular profile we guarantee that this is all that the app ever sees, i.e. we support isolation of the app runtime environment vs the distribution environment that the user is running. This is done by containers and namespaces where we mount only the system files and application files into the app namespace.
With this kind of isolation we guarantee two things, first of all there will never be any accidental leaks of dependencies. If your app depends on anything not in the supported platform it will just not run and the developer will immediately notice. Secondly, any incompatible changes in system libraries are handled and the app will still get a copy of the older versions when it runs.
One nice thing about this setup is that the application runtime environment will look like a minimal but standard linux install with the app and its dependencies in the standard prefixes like /usr/lib, etc. This means that when creating an application one can very easily reuse existing .deb or .rpm files by just extracting these and putting in the application bundle. Of course, it also will require a higher level of binary compatibility guarantees than what we have previously handled for modules the platform profiles provide. For instance, internal IPC protocols the platform libraries use absolutely have to be backwards compatible.
The sandbox model goes hand-in-hand with the isolation model of app deployment, in the sense that whatever the app should not be able to do it will not even see. So, for a fully sandboxed app we will not even mount in the users home directory in the app namespace, rather than not having access rights to it. (We will of course also offer applications with unlimited sandboxes so that existing apps can run.)
In order to talk with the sandboxes app we need a IPC model that handles the domain transition between the namespaces. This implies the kernel being involved, so we have been looking (again) at getting some form of dbus routing support into the kernel. Hopefully this will work out this time.
Unfortunately we also need IPC for the Xserver, which is very hard to secure. We’ve decided to just just ignore this for now however, as it turns out Wayland is a very good fit for this, since it naturally isolates the clients.
We also talked about implementing something similar to the Intents system in android as a way to allow sandboxed applications to communicate without necessarily knowing about each other. This essentially becomes a DBus service which keeps a registry of which apps implements the various interfaces we want to support (e.g. file picking, get-a-photo, share photo) and actually proxies the messages for these to the right destination. We had a long discussion about the name for these and came up with the name “Portals”, reflecting the domain-transition that these calls represent.
We’re continuing to discuss the details of the different parts and hopefully we can start implement parts of this soon. After the hackfest we will continue discussions on the gnome-os mailing list.