Technical Blog of Richard Hughes

Public Service Announcement: Debian helpfully defines datadir to be /usr/share/games for some packages, which means that the AppData and MetaInfo files get installed into /usr/share/games/appdata which isn’t picked up by the metadata parsers.

It’s probably safer to install the AppData files into $datarootdir/appdata as this will work even if a distro has redefined datadir to be something slightly odd. I’ve changed the examples on the AppData page, but if you maintain a game on Debian with AppData then this might affect you when Debian starts extracting AppSpream metadata in the next few weeks. Anyone affected will be getting email in the next few days, although it only looks to affect very few people.

Ever since we rolled out the GNOME Software Center, people have wanted to extend it to do other things. One thing that was very important to the Eclipse developers was a way of adding addons to the main application, which seems a sensible request. We wanted to make this generic enough so that it could be used in gedit and similar modular GNOME and KDE applications. We’ve deliberately not targeted Chrome or Firefox, as these applications will do a much better job compared to the package-centric operation of GNOME Software.

So. Do you maintain a plugin or extension that should be shown as an addon to an existing desktop application in the software center? If the answer is “no” you can probably stop reading, but otherwise, please create a file something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright 2014 Your Name Here <your@email.com> -->
<component type="addon">
<id>gedit-code-assistance</id>
<extends>gedit.desktop</extends>
<name>Code Assistance</name>
<summary>Code assistance for C, C++ and Objective-C</summary>
<url type="homepage">http://projects.gnome.org/gedit</url>
<metadata_license>CC0-1.0</metadata_license>
<project_license>GPL-3.0+</project_license>
<updatecontact>richard_at_hughsie.com</updatecontact>
</component>

This wants to be installed into /usr/share/appdata/gedit-code-assistance.metainfo.xml — this isn’t just another file format, this is the main component schema used internally by AppStream. Some notes when creating the file:

• You can use anything as the <id> but it needs to be unique and sensible and also match the .metainfo.xml filename prefix
• You can use appstream-util validate gedit-code-assistance.metainfo.xml if you install appstream-glib from git.
• Don’t put the application name you’re extending in the <name> or <summary> tags — so you’d use “Code Assistance” rather than “GEdit Code Assistance“
• You can omit the <url> if it’s the same as the upstream project
• You don’t need to create the metainfo.xml if the plugin is typically shipped in the same package as the application you’re extending
• Please use <_name> and <_summary> if you’re using intltool to translate either your desktop file or the existing appdata file and remember to add the file to POTFILES.in if you use one

Please grab me on IRC if you have any questions or concerns, or leave a comment here. Kalev is currently working on the GNOME Software UI side, and I only finished the metadata extractor for Fedora today, so don’t expect the feature to be visible until GNOME 3.14 and Fedora 21.

In the last few days, I’ve been asking people to create and ship AppData files upstream. I’ve:

• Sent 245 emails to upstream maintainers
• Opened 38 launchpad bugs
• Created 5 gnome.org bugs
• Opened 72 sourceforge feature requests
• Opened 138 github issues
• Created 8 bugs on Fedora trac
• Opened ~20 accounts on random issue trackers
• Used 17 “contact” forms

In doing this, I’ve visited over 600 upstream websites, helpfully identifying 28 that are stated as abandoned by thier maintainer (and thus removed from the metadata). I’ve also blacklisted quite a few things that are not actually applications and not suitable for the software center.

I’ve deliberately not included GNOME in this sweep, as a lot of the core GNOME applications already have AppData and most of the gnomies already know what to do. I also didn’t include XFCE appications, as XFCE has agreed to adopt AppData on the mailing list and are in the process of doing this already. KDE is just working out how to merge the various files created by Matthias, and I’ve not heard anything from LXDE or MATE. So, I only looked at projects not affiliated with any particular desktop.

For far, the response has been very positive, with at least 10% of the requests been actioned and some projects even doing new releases that I’ve been slowly uploading into Fedora. Another ~10% of requests are acknowlegdments from maintainers thay they would do this sometime before the next release. I have found a lot of genuinely interesting applications in my travels, and lot of junk. The junk is mostly unmaintained, and so my policy of not including applications (unless they have AppData manually added by the distro packager) that have not had an upstream release for the last 5 years seems to be valid.

At least 5 of the replies have been very negative, e.g. “how dare you ask me to do something — do it yourself” and things like “Please do not contact me again – I don’t want any new users“. The vast number of people have not responded yet — so I’m preparing myself for a deluge over the next few weeks from the people that care.

My long term aim is to only show applications in Fedora 22 with AppData, so it seemed only fair to contact the various upstream projects about an initiative they’re probably not familiar with. If we don’t get > 50% of applications in Fedora with the extra data we’ll have to reconsider such a strong stance. So far we’ve reached over 20%, which is pretty impressive for a standard I’ve been pushing for such a short amount of time.

So, if you’ve got an email from me, please read it and reply — thanks.

A few long months ago I asked everyone shipping a desktop application to also write an AppData file for the software installer. So far over 300 projects have written these files and there are over 500 upstream screenshots that have been taken. The number has been growing steadily, and most active projects now ship a file upstream. So, what do I want you to do now? :)

The original AppData specification had something like this:

<?xml version="1.0" encoding="UTF-8"?>
<application>
<id type="desktop">gnome-power-statistics.desktop</id>
<licence>CC0</licence>
...

This had a couple of problems. First was the spelling of license. I’m from Blightly, and forgot that I was supposed to be coding in en_US. The second was people frequently got confused that they were supposed to be specifying the license of that specific metadata file, rather than the license of the project as a whole. A few months ago we fixed this, and added the requirement of a copyright statement to please the Debian overlords:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright 2013 Richard Hughes <richard@hughsie.com> -->
<application>
<id type="desktop">gnome-power-statistics.desktop</id>
<metadata_license>CC0-1.0</metadata_license>
<project_license>GPL-2.0+ and GFDL-1.3</project_license>
...

The project licenses just have to be valid SPDX strings. You can use “ and ” and “ or ” or even brackets if required, just like in a spec file. The reason for standardising on SPDX is that it’s being used on lots of distros now, and we can also make the licence substrings clickable in gnome-software very easily.

So, if you’ve already written an AppData file please do three things:

• Make sure Copyright exists at the top of the file after the <?xml header
• Convert license into metadata_license and change to a SPDX ID
• Add project_license and add all the licenses used in your project.

In Fedora 21 I’m currently doing a mapping from License: in the spec file to the SPDX format, although it’s not a 1:1 mapping hence why I need this to be upstream. KDE is already shipping project_license in thier AppData files, but I’m not going to steal that thunder. Stay tuned.

So I did a bit more hacking on PackageKit, appstream-glib and gnome-software last night. We’ve now got screenshots from Debian (which are not very good) and long application descriptions from the package descriptions (which are also not very good). It works well enough now, although you now need PackageKit from master as well as appstream-glib and gnome-software.

This is my last day of hacking on the Ubuntu version, but I’m hopeful other people can take what I’ve done and continue to polish the application so it works as well as it does on Fedora. Tasks left to do include:

• Get aptcc to honour the DOWNLOADED filter flag so we can show applications in the ‘Updates’ pane
• Get aptcc to respect the APPLICATION filter to speed up getting the installed list by an order of magnitude
• Get gnome-software (or appstream-glib) to use the system stock icons rather than the shitty ones shipped in the app-install-data package
• Find out a way to load localized names and descriptions from the app-install-data gettext archive and add support to appstream-glib. You’ll likely need to call dgettext(), bindtextdomain() and bind_textdomain_codeset()
• Find out a way how to populate the ‘quality’ stars in gnome-software, which might actually mean adding more data to the app-install desktop files. This is kind of data we need.
• Find out why aptcc sometimes includes the package summary in the licence detail position
• Improve the package details to human readable code to save bullet points and convert to a UTF-8 dot
• Get the systemd offline-updates code working, which is completely untested
• Find out why aptcc seems to use a SHA1 hash for the repo name (e.g. pkcon repo-list)
• Find out why aptcc does not set the data part of the package-id to be prefixed with installed: for installed packages

If you can help with any of this, please grab me on #PackageKit on freenode.

After an afternoon of hacking on appstream-glib, I can show the fruits of my labours:

This needs gnome-software and appstream-glib from git master (or gnome-apps-3.14 in jhbuild) and you need to manually run PackageKit with the aptcc backend (--enable-aptcc).

It all kinda works with the data from /usr/share/app-install/*, but the icons are ugly as they are included in all kinds of sizes and formats, and also there’s no long descriptions except for the two (!) installed applications new enough to ship local AppData files.Also, rendering all those svgz files is muuuuch slower than a pre-processed png file like we ship with AppStream. The installed view also seems not to work. Only the C locale is present too, as I’ve not worked out how to get all the translations from an external gettext file in appstream-glib. I’d love to know how the Ubuntu software center gets long descriptions and screenshots also. But it kinda works. Thanks.

Today I released gnome-software 3.12.0 — with a number of new features and a huge number of bugfixes:

I think I’ve found something interesting to install — notice the auto-generated star rating which tells me how integrated the application is with my environment (i.e. is it available in my language) and if the application is being updated upstream. Those thumbnails look inviting:

We can continue browsing while the application installs — also notice the ‘tick’ — this will allow me to create and modify application folders in gnome-shell so I can put the game wherever I like:

The updates tab looks a little sad; there’s no update metadata on rawhide for my F20 GNOME 3.12 COPR, but this looks a lot more impressive on F20 or the yet-to-be-released F21. At the moment we’re using the AppData metadata in place of update descriptions there. Yet another reason to ship an AppData file.

We can now safely remove sources, which means removing the applications and addons that we installed from them. We don’t want applications sitting around on our computer not being updated and causing dependency problems in the future.

Development in master is now open, and we’ve already merged several large patches. The move to libappstream-glib is a nice speed boost, and other more user-visible features are planned. We also need some documentation; if you’re interested please let us know!

Quite a few people have asked me how the AppStream distro metadata is actually generated for thier app. The actual extraction process isn’t trivial, and on Fedora we also do things like supply missing AppData files for some key apps, and replacing some upstream screenshots on others.

In order to make this more transparent, I’m going to be uploading the logs of each generation run. If you’ve got a few minutes I’d appreciate you finding your application there and checking for any warnings or errors. The directory names are actually Fedora package names, but usually it’s either 1:1 or fairly predictable.

If you’ve got a application that’s being blacklisted when it shouldn’t be, or a GUI application that’s in Fedora but not in that list then please send me email or grab me on IRC. The rules for inclusion are here. Thanks.

For a few years now Appstream and AppData adoption has been growing. We’ve got client applications like GNOME Software consuming the XML files, and we’ve got several implementations of metadata generators for a few distros now. We’ve also got validation tools we’re encouraging upstream applications to use.

The upshot of this was the same code was being duplicated across 3 different projects of mine, all with different namespaces and slightly different defined names. Untangling this mess took a good chunk of last week, and I’ve factored out 2759 lines of code from gnome-software, 4241 lines from createrepo_as, and the slightly less impressive 178 lines from appdata-tools.

The new library has a simple homepage, and so far a single release. I’d encourage people to check this out and provide early comments, as as soon as gnome-software branches for 3-12 I’m going to switch it to using this. I’m also planning on switching createrepo_as and and appdata-tools for the next releases too so things like jhbuild modulesets need to be updated and tested by somebody.

Appstream-Glib 0.1.0 provides just enough API to make sense for a first release, but I’m going to be continuing to abstract out useful functionality from the other projects to share even more code. I’ve spent a few long nights profiling the XML parsing code, and I’m pleased to say the load time of gnome-software is 160ms faster with this new library, and createrepo_as completes the metadata generation 4 minutes faster. Comments, suggestions and patches very welcome. There’s a Fedora package linked from the package review bug if you’d rather test that. Thanks.

I’ve finished building the packages for GNOME 3.11.90. I’ve done this as a Fedora 20 COPR. It’s probably a really good idea to test this in a VM rather than your production systems as it’s only had a small amount of testing.

If it breaks, you get to keep all 132 pieces. It’s probably also not a good idea to be asking fedora-devel or fedoraforums for help when using these packages. If you don’t know how to install a yum repo these packages are not for you.

Comments and suggestions, welcome. Thanks.