Offline Updates in Fedora 20

In GNOME 3.10 we’re encouraging more people to use the offline-update functionality which we’ve been using in Fedora for a little while now. A couple of people have told me it’s really slow, but I hadn’t seen an offline update take more than a minute or so as I test updates all the time. To reproduce this, I spun up a seldom-used Fedora 20 alpha image and let GNOME download and prepare all the updates in the background. I then added some profiling code to the pk-offline-update binary, and rebooted. The offline update took almost 17 minutes to run.

So, what was it doing all that time, considering that we’ve already downloaded the packages and depsolved the transaction:

Transaction Phase Time (s)
Start up PackageKit 0.3
Starting up yum 3
Depsolving 10
Signature Check 8
Test Commit 5
Install new packages 704
Remove old packages 168
Run post-install scripts 90

This is about an order of magnitude slower than what I expected. Some of my observations:

  • 10 seconds to depsolve an already depsolved transaction
  • 8 seconds to check a few hundred signatures
  • 168 seconds just to delete a few thousand files
  • over 10 minutes to install a few hundred RPMs seems crazy
  • 90 seconds to rebuild a few indexes seems like a huge amount of time

Some notable offenders:

Package Time to install (s)
selinux-policy-targeted 122
kernel-devel 25
libreoffice-core 21
selinux-policy 17
hugin 12

 

Package Time to cleanup (s)
gramps 11
wireshark-gnome 8
hugin 7
meld 6
control-center 5

Hopefully Fedora 21 will move to the hawkey backend, and we can get closer to raw librpm speed (which seems to be quite a speed boost) but even that is too slow. I’ll be looking into the individual packages this week, and trying to find what makes them so slow, and what we can do about them to speed things up.

GNOME Shell and GNOME Software

The ever-awesome Matthias Clasen added a nice feature to GNOME Software a couple of weeks ago:
gnome-software-shell-search
It’ll be available in GNOME 3.12 in a few months time.

Upstream adoption of AppData so far

By popular request, some update on the upstream adoption of AppData so far:

Applications in Fedora with long descriptions: 168 (9%)
Applications in Fedora with screenshots: 140 (7%)
Applications in GNOME with AppData: 60 (50%)
Applications in KDE with AppData: 1 (1%)
Applications in XFCE with AppData: 0 (0%)

You can look at a few ways:

  • We’ve made significant progress in the last year-or-so and many popular applications are already shipping the extra data.
  • There are a lot of situations where the upstream authors do not know what an AppData file is, don’t have time to add one, or simply do not care.
  • GNOME is clearly ahead of KDE and XFCE, probably because of the existing GNOME Goal and my nag emails to the desktop-devel mailing list. A little thing to bear in mind is that Apper (the KDE application installer) can also make use of the AppStream data, so this is a little disappointing for KDE users who probably don’t see any difference at the moment.

So where do we go from here? Clearly KDE and XFCE have some catching up to do, and I need someone familiar with those communities to lead this effort. There is also a huge number of upstreams that need a little push in the right direction, and I’ve been trying to do that for the last couple of months. Without help, this would be a never-ending battle for me. A little reminder: In GNOME 3.12 we are penalising applications that don’t ship AppData by including them lower in the search results, and in GNOME 3.14 we’re not going to be showing them at all.

If you’re interested to see all the applications shown by default in Fedora 20, I’ve put together this page showing a quick overview. If you see anything there that shouldn’t be an application and needs blacklisting, just let me know. If you see an application you care about without a long description or screenshots, then please file a bug upstream pointing them at the AppData specification page. Thanks.

How to generate AppStream metadata for Fedora

I’m generating all the Fedora AppStream metadata by hand at the moment. Long term this is going to move to koji, but since we’re still tweaking the generator, adding features and fixing bugs it seems too early to fully integrate things. This is fine if you just care about the official Fedora sources, but a lot of people want to use applications from other less, ahem, free repos.

If you manage a repository and want to generate AppStream metadata yourself it’s really quite easy if you follow these instructions, although building the metadata can take a long time. Lets assume you run a site called MegaRpms and you want to target Fedora 20.

First, checkout the latest version of fedora-appstream and create somewhere we can store all the temporary files. You’ll want to do this on a SSD if possible.

$ mkdir megarpms
$ cd megarpms

Then create a project file with all the right settings for your repo. Lets assume you have two seporate trees, ‘megarpms’ and ‘meagarpms-updates’.

$ cat project.conf
[AppstreamProject]
DistroTag=f20
RepoIds=megarpms,megarpms-updates
DistroName=megarpms-20
ScreenshotMirrorUrl=http://www.megarpms.org/screenshots/

The screenshot mirror URL is required if you want to be able to host screenshots for applications. If you don’t want to (or can’t afford the hosting costs) then you can comment this out and no screenshots will be generated.

Then we can actually download the packages we need to extract. Ensure that both megarpms and megarpms-updates are enabled in /etc/yum.conf.d/ and then start downloading:

$ sudo ../fedora-download-cache.py

This requires root as it uses and updates the system metadata to avoid duplicating the caches you’ve probably already got. After all the interesting packages are downloaded you can do:

$ ../fedora-build-all.py

Now, go and make a cup of tea and wait patiently if you have a lot of packages to process. After this is complete you can do:

$ ../fedora-compose.py

This spits out megarpms-20.xml.gz and megarpms-20-icons.tar.gz — and you now have two choices what to do with these files. You can either upload them with the rest of the metadata you ship (e.g. in the same directory as repomd.xml and primary.sqlite.bz2) which will work with Fedora 21 and higher.
For Fedora 20, you have to actually install these files, so you can do something like this in the megarpms-release.spec file:

Source1: http://www.megarpms.org/temp/megarpms-20.xml.gz
Source2: http://www.megarpms.org/temp/megarpms-20-icons.tar.gz
mkdir -p %{buildroot}%{_datadir}/app-info/xmls
cp %{SOURCE1} %{buildroot}%{_datadir}/app-info/xmls
mkdir -p %{buildroot}%{_datadir}/app-info/icons/megarpms-20
tar xvzf %{SOURCE2}
cd -

This ensures that gnome-software can access both data files when starting up. If you have any other questions, concerns or patches, please get in touch. This is all very Fedora specific (rpm files, Yum API, various hardcoded package names) but if you’re interested in using fedora-appstream on your distro and want to actually do the work I’d welcome patches to make it less fedora-centric. SUSE generates the AppStream files in a completely different way.

PackageKit service packs and catalogs

Does anyone actually use the PackageKit service pack or catalog functionality? If there are no users I’m intending to rip out the unused and unloved features from GNOME 3.12. Please say now, or forever hold your peace. Thanks.

How to take 16:9 Screenshots

A few people contacted me after discovering that screenshots should be taken in a 16:9 aspect ratio. The question was basically, how do I do that?

After trying to do this myself for my applications, I too discovered it’s hard. The wmctrl command doesn’t seem to play very nicely with CSD, and certainly won’t work in wayland. So, Owen Taylor (GNOME hacker extraordinaire) to the rescue.

Owen wrote a GNOME Shell extension (which I’ve modified a little) which resizes the current window to a 16:9 size when you press Ctrl+Alt+S. If you press it again, the window will get larger, to the next recommended 16:9 size. Press Ctrl+Alt+Shift+S and the window will get smaller to the previous size. Get the code on github or on extensions.gnome.org. I probably need some more testing as well. Patches very welcome if you’re good at shell extensions :)

With this extension installed I was able to screenshot all my applications that I maintain in no time at all. I’ve been uploading the screenshots into ${projects}/data/appdata and referencing the git.gnome.org URL in the AppData file so anyone can easily update them when the code changes, but this is completely up to you.

AppData Validation Tool Update 2

There’s a new version of the AppData validation tool, and this one actually downloads and validates the screenshots you’ve included in the .appdata.xml files. See the AppData specifications for guidelines on how to make good screenshots.

Get it while it’s hot, appdata-tools-0.1.4.tar.xz or I’ve built Fedora RPMs too.

Please yell if you think you’ve written a valid file, and it fails to validate using the tool. We’ve added a lot more checks, and we’re getting more strict — so please revalidate your file if you’ve already done so. Thanks!

Copyright in AppData files

More AppData news! I’ve been contacted by someone connected to Debian Legal, who apparently want me to add copyright information to the AppData files for licence compliance. Whilst most of the files are CC0 (basically public domain) it doesn’t seem super important, but it is technically required. If you ship an AppData file, I’d appreciate it if you could add <!-- Copyright 2013 First Lastname --> on the second line in the file. I’ve updated the AppData specification examples accordingly.

Additionally, the appdata-validate tool will warning you about missing copyright information if you use the –strict command line argument in the next release.

Frikin’ Awesome Apps (without AppData)

In GNOME Software, we show a list of applications for each category that we think are frikin’ awesome. Some have AppData, and some don’t. For the ones that don’t yet have AppData it leaves the responsibility of writing the long description to the Linux community, where we can push the data back to upstream so that all the distributions can benefit. So far we’ve had a superb reaction from lots of upstream projects.

For Fedora 20 we want all the awesome apps to have AppData, so users can evaluate the application before installing it. It would add a really nice bit of polish to the whole experience. If you can spare 5 minutes and want to help, I’ve got another shared document here.

The list of awesome-but-unloved apps is: audacity, ardour2, gnome-banshee, rosegarden, sound-juicer, doom, openarena, xonotic, tremulous, btanks, frozen-bubble, quadrapassel, neverball, gnomine, wesnoth, supertuxkart, redeclipse, lyx, gparted, virt-manager, eclipse, gitg, monodevelop, blender, shotwell, octave, saoimage, workrave, celestia, polari, pidgin, chromium, pitivi, vlc and openshot. If you want to add any other apps there, please feel free, although I would much prefer the AppData file to go upstream if you’re a contributor or maintainer of a project. Remember, we can’t feature applications on the front page that don’t have AppData, so that’s an added incentive. I’d love to expose more people to blender for instance.

There’s no way I can do all this myself for Fedora 20, so I really need your help. Again. Thanks.

GNOME Goal: AppData for all applications

Daniel Mustieles García has proposed a GNOME Goal that every official application in GNOME should ship an AppData file. See the wiki page and check your module. Loads of modules have already added files, but remember, the AppData file has to be translated and validated before it’s ‘done‘ for the GNOME Goal.

Thanks!

Bad Behavior has blocked 2769 access attempts in the last 7 days.