Upgrading Fedora 23 to 24 using GNOME Software

I’ve spent the last couple of days fixing up all the upgrade bugs in GNOME Software and backporting them to gnome-3-20. The idea is that we backport gnome-software plus a couple of the deps into Fedora 23 so that we can offer a 100% GUI upgrade experience. It’s the first time we’ve officially transplanted a n+1 GNOME component into an older release (ignoring my unofficial Fedora 20 whole-desktop backport COPR) and so we’re carefully testing for regressions and new bugs.

If you do want to test upgrading from F23 to F24, first make sure you’ve backed up your system. Then, install and enable this COPR and update gnome-software. This should also install a new libhif, libappstream-glib, json-glib and PackageKit and a few other bits. If you’ve not done the update offline using [the old] GNOME Software, you’ll need to reboot at this stage as well.

Fire up the new gnome-software and look at the new UI. Actually, there’s not a lot new to see as we’ve left new features like the ODRS reviewing service and xdg-app as F24-only features, so it should be mostly the same as before but with better search results. Now go to the Updates page which will show any updates you have pending, and it will also download the list of possible distro upgrades to your home directory.

As we’re testing upgrading to a pre-release, we have to convince gnome-software that we’re living in the future. First, open ~/.cache/gnome-software/3.20/upgrades/fedora.json and search for f24. Carefully change the Under Development string to Active then save the file. Log out, back in and the launch gnome-software again or wait for the notification from the shell. If all has gone well you should see a banner telling you about the new upgrade. If you click Download go and get a coffee and start baking a cake, as it’s going to take a long time to download all that new goodness. Once complete just click Install, which prompts a reboot where the packages will be installed. For this step you’ll probably want to bake another cake. We’re not quite in an atomic instant-apply world yet, although I’ll be talking a lot more about that for Fedora 25.

With a bit of luck, after 30 minutes staring at a progressbar the computer should reboot itself into a fresh new Fedora 24 beta installation. Success!


If you spot any problems or encounter any bugs, please let me know either in bugzilla, email or or IRC. I’ve not backported all the custom CSS for the upgrade banner just yet, but this should be working soon. Thanks!

Age Ratings in GNOME Software: Introducing OARS?

In GNOME Software we show lots of applications ranging from games aimed at pre-schoolers to applications explicitly designed to download, well, porn. A concept that is fairly well understood by parents is age ratings, and there are well known and trusted ratings bodies such as the ESRB and PEGI, as well as other country-specific schemes. Parents can use the ratings to control what kind of content is available to install, and vendors can use the ratings as a legal (or common-sense) control who gets to purchase what.

The ratings systems between countries are varied, varying from descriptions such as “M” which will be familiar for US users, “R” for Australian users to the slightly more obvious “18+” rating for European users. The differing ratings authorities define what is allowed in each category in slightly different way, some allowing mild profanity for a “7+” rating, and others none at all. Some countries consider drug taking in a video game to be no more dangerous as to mild cursing, other countries consider this on the same level as sexual violence.


So, we’re sunk, right? Nearly. There exists a group called “International Age Rating Coalition” which allows developers to register (sometimes for free), answer a simple questionnaire and out pops the ratings they should use for various countries. The IARC is made up of the regulatory bodies all over the planet, and so you can use the actual trademarked age rating images for your product. ish.

If you want to build a a software center, say GNOME Software for example, you have to pay a license fee. A $100,000 annual fee, plus extra per application shown in the software center. This is prohibitive for us, and would mean we couldn’t have the same functionality in other software center interfaces.

We could easily provide in the AppData files details about the application/game. This can be combined with a rule engine specific to the country of viewing, which would pop out a rating. I think the ESRB would be hard pushed to trademark “M” as an age rating, although I completely agree they have correctly and sensibly trademarked the stylized logo for the PG rating, along with the “ESRB” name itself. I don’t think this should stop us using an “PG” or “M” rating in the software center as long as we avoid these trademarks and copyrights.

I’m happy to work on a new system to both generate the AppData upstream information from a questionaire, and the rule engines that processes these rules and pops out a rating. The question then becomes, is this useful? Is this something that people would actually want? Comments welcome.

p.s. OARS: “Open Age Rating System”, name is work in process.

Moderate reviews in GNOME Software

I’m pondering adding something like this for GNOME Software:

Screenshot from 2016-02-11 20-31-28

The idea is it would be launched using a special geeky-user-only gnome-software --mode=moderate command line, so that someone can up or down-vote any reviews that are compatible with their country code. This would be used by super-users using community distros like Fedora, and perhaps people in a QA team for distros like RHEL. The reason I wanted to try to use gnome-software rather than just doing it in a web-app was that you have access to the user machine hash, so you can hide reviews the user has already voted on without requiring a username and password that has to be matched to the local machine. We also have all the widgets and code in place, so it is really just a couple of hundred lines of code for this sekret panel. The server keeps track of who voted on what and so reviewers can just close the app, open it a few weeks later and just continue only moderating the reviews that came in since then.

I can’t imagine the tool would be used by many people, but it does make reviewing comments easy. Comments welcome.

Comments are live

With a huge amount of help from Robert Ancell for a lot of the foundations for the new code, I’ve pushed a plugin today to allow anonymous rating of applications.

Screenshot from 2016-02-10 17-16-04

If people abuse or spam this I’ll take the feature away until we can have OpenID logins in GNOME Online Accounts, but I’m kinda hoping people won’t be evil. The server is live and accepting reviews and votes, but the API isn’t set in stone.

Anonymous reviews in GNOME Software

Choosing an application to install is hard when there are lots of possible projects matching a specific search term. We already list applications based on the integration level and with useful metrics like “is it translated in my language” and this makes sure that high quality applications are listed near the top of the results. For more information about an application we often want a more balanced view than the PR speak or unfounded claims of the upstream project. This is where user-contributed reviews come in.


To get a user to contribute a review (which takes time) we need to make the process as easy as possible. Making the user create a user account on yet-another-webservice will make this much harder and increase the barrier to participation to the point that very few people would contribute reviews. If anonymous reviewing does not work the plan is to use some kind of attestation service so you can use a GMail or Facebook for confirming your identity. At this point I’m hoping people will just be nice to each other and not abuse the service although this reviewing facility will go away if it starts being misused.

Designing an anonymous service is hard when you have to be resilient against a socially awkward programmer with specific political ideologies. If you don’t know any people that match this description you have obviously never been subscribed to fedora-devel or memo-list.

Obviously when contacting a web service you share your IP address. This isn’t enough to uniquely identify a machine and user, which we want for the following reasons:

  • Allowing users to retract only their own reviews
  • Stopping users up or down-voting the same review multiple times

A compromise would be to send a hash of two things that identify the user and machine. In GNOME Software we’re using a SHA1 hash of the machine-id and the UNIX username along with a salt, although this “user_id” is only specified as a string and the format is not checked.

For projects like RHEL where we care very much what comments are shown to paying customers we definitely want reviews to be pre-approved and checked before showing to customers. For distros like Fedora we don’t have this luxury and so we’re going to rely on the community to self-regulate reviews. Reviews are either up-voted or down-voted according how useful they are along with the nuclear option of marking the review as abusive.


By specifying the users current locale we can sort the potential application reviews according to a heuristic that we’re still working on. Generally we want to prefer useful reviews in the users locale and hide ones that have been marked as abusive, and we also want to indicate the users self-review so they can remove it later if required. We also want to prioritize reviews for the current application version compared to really old versions of these applications.

Comments welcome!

AppData and the gettext domain

When users are searching for software in GNOME Software it is very important to answer the the question “Is this localized in my language?” If you can only speak Swedish then an application talking just in American English is not much use at all. The way we calculate this in the AppStream builder is to look at the compiled .mo files, breaking them apart and then using statistics to work out what locales are included.

When we’re processing distro packages we usually extract them one at a time. We first try for a gettext domain (the .mo file name) of the distro package name, and if that’s not found then we just try and find the first .mo file in any of the locale directories. This works about 70% of the time (which is good) but fails about 30% of the time (which is bad). For xdg-app we build the application in a special prefix, along with any dependent libraries. We don’t have a distro package name for the bundle (only the application ID) and so the “first .mo file we can find” heuristic fails more often that it works. We clearly need some more information about the gettext domain from the upstream project.

AppData to the rescue. By adding this in the AppData file informs the AppStream generation code in the xdg-app builder what gettext domain to use for an application. To use this you just need to add:

  <translation type="gettext">the_gettext_domain_here</translation>

under the <component> tag. The gettext domain is normally set in the configure.ac file with the GETTEXT_PACKAGE define. If you don’t have this extra data in your application then appstream-util validate is soon going to fail, and your application isn’t going to get the language metadata and so will be lower in the search results for users using GNOME Software in a non-C locale. If your GNOME application is available in jhbuild the good news is that I’ve automatically added the <translation> tag to 104 projects semi-automatically today. For XFCE and KDE I’m going to be sending emails to the development mailing lists tomorrow. For all other applications I’m going to be using the <update_contact> email address set in the AppData file for another mass-emailing.

Although it seems I’m asking people to do more things again and again I can assure you that slowly we’re putting the foundations in place for an awesome software installer experience. Just today I merged in the xdg-app branch into gnome-software and so I’m hoping to have a per-user xdg-app preview available in Fedora 24. Exciting times. :)

The importance of Keywords for the software center

In the software center we allow the user to search using case-insensitive keywords, for instance searching for ‘excel’ could match Libreoffice Calc or many other free software spreadsheet applications. At the moment we use the translated keywords set in the desktop file, any extra <keyword> entries in the AppData file, and then fall back to generating tokens from the name, summary and description using a heuristic. This heuristic works most of the time, but a human can often do much better when we know what the most important words are. I’ve started emailing maintainers who do not have any keywords in their application (using the <update_contact> details in the AppData file), but figured I should also write something here.

So, what do I want you to do? If you have no existing keywords, I would like you to add some keywords in the desktop file or the AppData file. If you want the keywords to be used by GNOME Shell as well (which you probably do), the best place to put any search terms is in the keywords section of the desktop file. This can also be marked as translatable so non-English users can search in their own language. This would look something like Keywords=3D;printer; (remember the trailing semicolon!)

The alternative is to put the keywords in the AppData file so that they are only used by the software center and not the desktop shell. You can of course combine putting keywords in both places. The AppData keywords can also be translated, and would look like this:


Of course, you don’t have to do a release with this fix straight away, and if you have a stable branch it would be a good thing to backport this as well if it does not add translated strings or you have no string freeze policy. Nothing bad will happen if you ignore this request, but please be aware that matches from keywords are ordered higher in the search results than other partial matches from the name or summary. You also don’t have to add keywords that are the same as the application name or package name, as these are automatically added as case insensitive search tokens. If you don’t have any keywords then your application will still be visible in the various software centers, but it may be harder to find.

Comments welcome.

GNOME Software and xdg-app

Here’s a little Christmas present. This is GNOME Software working with xdg-app to allow live updates of apps and runtimes.

Screenshot from 2015-12-22 15-06-44

This is very much a prototype and needs a lot more work, but seems to work for me with xdg-app from git master (compile with --enable-libxdgapp). If any upstream projects needed any more encouragement, not including an AppData file means the application gets marked as nonfree as we don’t have any project licensing information. Inkscape, I’m looking at you.

The Linux Vendor Firmware Service Welcomes Dell

I’m finally able to talk about one of the large vendors who have been trialing the LVFS service for the last few months. Dell have been uploading embargoed UEFI firmware files with metadata for a while, testing the process and the workflow ready for upcoming new models. Mario (Dell) and myself (Red Hat) have been working on fixing all the issues that pop up on real hardware and making the web service both secure and easy to use.

Screenshot from 2015-12-10 08-43-08

The Dell Edge Gateway will be available for purchase soon. When it goes on sale, firmware updates in Linux will work out-of-the-box. I’ve been told that Dell are considering expanding out the LVFS support to all new models supporting UEFI updates. In order to prioritize what models to work on first, I’ve been asked to share this anonymous survey on what Dell hardware people are using on Linux and to gauge if people actually care about being able to upgrade the firmware easily in Linux.

In November, 224 firmware files were installed onto client systems using fwupd. At the moment to update the firmware metadata you need to manually click the refresh button in the updates page, which so far 40,000 people have done. Given that the ColorHug hardware is the only released hardware with firmware on the LVFS, the 224 downloads is about what I expected. When we have major vendors like Dell (and other vendors I can’t talk about yet) shipping real consumer hardware with UEFI update capability the number of files provided should go up by orders of magnitude.

For Fedora 24 we’ll be downloading the firmware metadata automatically (rather than requiring a manual refresh in the updates panel) and we’ve been using the Fedora 23 users as a good way of optimizing the service so we know we can handle the load when we get hundreds of thousands of automatic requests a month. Fedora 24 will also be the first release able to do updates on DFU USB devices, and also the first release with system upgrade capabilities inside GNOME Software so it’s quite exciting from my point of view.

With Dell on board, I’m hoping it will give some of the other vendors enough confidence in the LVFS to talk about distributing their own firmware in public. The LVFS is something I run for all distributions free of charge, but of course Red Hat pays for my time to develop and run the service. I’m looking forward to working with more Red Hat partners and OpenHardware vendors adding even more firmware for even more types of device in the future.