Here’s a little timeline of some fun we had with the GNOME master Flatpak runtime last week:
- Tuesday, July 10: a bad runtime build is published. Trying to start any application results in
error while loading shared libraries: libdw.so.1: cannot open shared object file: No such file or directory.Problem is the library is present in org.gnome.Sdk instead of org.gnome.Platform, where it is required. - Thursday, July 12: the bug is reported on WebKit Bugzilla (since it broke Epiphany Technology Preview)
- Saturday, July 14: having returned from GUADEC, I notice the bug report and bisect the issue to a particular runtime build. Mathieu Bridon fixes the issue in the freedesktop SDK and opens a merge request.
- Monday, July 16: Mathieu’s fix is committed. We now have to wait until Tuesday for the next build.
- Tuesday, Wednesday, and Thursday: we deal with various runtime build failures. Each day, we get a new build log and try to fix whatever build failure is reported. Then, we wait until the next day and see what the next failure is. (I’m not aware of any way to build the runtime locally. No doubt it’s possible somehow, but there are no instructions for doing so.)
- Friday, July 20: we wait. The build has succeeded and the log indicates the build has been published, but it’s not yet available via
flatpak update - Saturday, July 21: the successful build is now available. The problem is fixed.
As far as I know, it was not possible to run any nightly applications during this two week period, except developer applications like Builder that depend on org.gnome.Sdk instead of the normal org.gnome.Platform. If you used Epiphany Technology Preview and wanted a functioning web browser, you had to run arcane commands to revert to the last good runtime version.
This multi-week response time is fairly typical for us. We need to improve our workflow somehow. It would be nice to be able to immediately revert to the last good build once a problem has been identified, for instance.
Meanwhile, even when the runtime is working fine, some apps have been broken for months without anyone noticing or caring. Perhaps it’s time for a rethink on how we handle nightly apps. It seems likely that only a few apps, like Builder and Epiphany, are actually being regularly used. The release team has some hazy future plans to take over responsibility for the nightly apps (but we have to take over the runtimes first, since those are more important), and we’ll need to somehow avoid these issues when we do so. Having some form of notifications for failed builds would be a good first step.
P.S. To avoid any possible misunderstandings: the client-side Flatpak technology itself is very good. It’s only the server-side infrastructure that is problematic here. Clearly we have a lot to fix, but it won’t require any changes in Flatpak.
I don’t disagree that the current setup is shit, especially when I’m not around, and we need to fix it. However, you should be able to build the gnome runtime locally by just checking out gnome-sdk-images and running make.
Since I’m mentioned here…
I know I can build locally, and how to do it. I just don’t bother, and directly commit/push fixes, because it’s not faster to do a local build on my laptop than to wait for the next failure/success on the build server.
Also, as We build far less on the gnome builder these days i just bumped it to build everything twice a day. I also charged the import cron script on sdk.gnome.org to run every hour instead of once a day. However it would be best if that change was made in puppet instead or it will be reverted eventually.
> It seems likely that only a few apps, like Builder and Epiphany, are actually being regularly used.
Honestly, this shouldn’t be a surprise – most users want stability, and have no interest in nightly builds. The people who might use nightlies are generally going be people closely associated with the project and choose to spend time doing testing and reporting problems…
Thank you for fixing the issue quickly. I’m now reading your blog with the latest Epiphany Technology Preview again, which is my favorite browser for dogfooding of WebKitGTK+.