Technical Blog of Richard Hughes

Choosing an application to install is hard when there are lots of possible projects matching a specific search term. We already list applications based on the integration level and with useful metrics like “is it translated in my language” and this makes sure that high quality applications are listed near the top of the results. For more information about an application we often want a more balanced view than the PR speak or unfounded claims of the upstream project. This is where user-contributed reviews come in.

To get a user to contribute a review (which takes time) we need to make the process as easy as possible. Making the user create a user account on yet-another-webservice will make this much harder and increase the barrier to participation to the point that very few people would contribute reviews. If anonymous reviewing does not work the plan is to use some kind of attestation service so you can use a GMail or Facebook for confirming your identity. At this point I’m hoping people will just be nice to each other and not abuse the service although this reviewing facility will go away if it starts being misused.

Designing an anonymous service is hard when you have to be resilient against a socially awkward programmer with specific political ideologies. If you don’t know any people that match this description you have obviously never been subscribed to fedora-devel or memo-list.

Obviously when contacting a web service you share your IP address. This isn’t enough to uniquely identify a machine and user, which we want for the following reasons:

• Allowing users to retract only their own reviews
• Stopping users up or down-voting the same review multiple times

A compromise would be to send a hash of two things that identify the user and machine. In GNOME Software we’re using a SHA1 hash of the machine-id and the UNIX username along with a salt, although this “user_id” is only specified as a string and the format is not checked.

For projects like RHEL where we care very much what comments are shown to paying customers we definitely want reviews to be pre-approved and checked before showing to customers. For distros like Fedora we don’t have this luxury and so we’re going to rely on the community to self-regulate reviews. Reviews are either up-voted or down-voted according how useful they are along with the nuclear option of marking the review as abusive.

By specifying the users current locale we can sort the potential application reviews according to a heuristic that we’re still working on. Generally we want to prefer useful reviews in the users locale and hide ones that have been marked as abusive, and we also want to indicate the users self-review so they can remove it later if required. We also want to prioritize reviews for the current application version compared to really old versions of these applications.

Comments welcome!

When users are searching for software in GNOME Software it is very important to answer the the question “Is this localized in my language?” If you can only speak Swedish then an application talking just in American English is not much use at all. The way we calculate this in the AppStream builder is to look at the compiled .mo files, breaking them apart and then using statistics to work out what locales are included.

When we’re processing distro packages we usually extract them one at a time. We first try for a gettext domain (the .mo file name) of the distro package name, and if that’s not found then we just try and find the first .mo file in any of the locale directories. This works about 70% of the time (which is good) but fails about 30% of the time (which is bad). For xdg-app we build the application in a special prefix, along with any dependent libraries. We don’t have a distro package name for the bundle (only the application ID) and so the “first .mo file we can find” heuristic fails more often that it works. We clearly need some more information about the gettext domain from the upstream project.

AppData to the rescue. By adding this in the AppData file informs the AppStream generation code in the xdg-app builder what gettext domain to use for an application. To use this you just need to add:

  <translation type="gettext">the_gettext_domain_here</translation>

under the <component> tag. The gettext domain is normally set in the configure.ac file with the GETTEXT_PACKAGE define. If you don’t have this extra data in your application then appstream-util validate is soon going to fail, and your application isn’t going to get the language metadata and so will be lower in the search results for users using GNOME Software in a non-C locale. If your GNOME application is available in jhbuild the good news is that I’ve automatically added the <translation> tag to 104 projects semi-automatically today. For XFCE and KDE I’m going to be sending emails to the development mailing lists tomorrow. For all other applications I’m going to be using the <update_contact> email address set in the AppData file for another mass-emailing.

Although it seems I’m asking people to do more things again and again I can assure you that slowly we’re putting the foundations in place for an awesome software installer experience. Just today I merged in the xdg-app branch into gnome-software and so I’m hoping to have a per-user xdg-app preview available in Fedora 24. Exciting times. :)

With a huge amount of support from alex, installing applications using xdg-app in GNOME Software is working. There are still a few rough edges, but it’s getting there quickly now.

In the software center we allow the user to search using case-insensitive keywords, for instance searching for ‘excel’ could match Libreoffice Calc or many other free software spreadsheet applications. At the moment we use the translated keywords set in the desktop file, any extra <keyword> entries in the AppData file, and then fall back to generating tokens from the name, summary and description using a heuristic. This heuristic works most of the time, but a human can often do much better when we know what the most important words are. I’ve started emailing maintainers who do not have any keywords in their application (using the <update_contact> details in the AppData file), but figured I should also write something here.

So, what do I want you to do? If you have no existing keywords, I would like you to add some keywords in the desktop file or the AppData file. If you want the keywords to be used by GNOME Shell as well (which you probably do), the best place to put any search terms is in the keywords section of the desktop file. This can also be marked as translatable so non-English users can search in their own language. This would look something like Keywords=3D;printer; (remember the trailing semicolon!)

The alternative is to put the keywords in the AppData file so that they are only used by the software center and not the desktop shell. You can of course combine putting keywords in both places. The AppData keywords can also be translated, and would look like this:

<keywords>
<keyword>3D</keyword>
<keyword>printer</keyword>
</keywords>

Of course, you don’t have to do a release with this fix straight away, and if you have a stable branch it would be a good thing to backport this as well if it does not add translated strings or you have no string freeze policy. Nothing bad will happen if you ignore this request, but please be aware that matches from keywords are ordered higher in the search results than other partial matches from the name or summary. You also don’t have to add keywords that are the same as the application name or package name, as these are automatically added as case insensitive search tokens. If you don’t have any keywords then your application will still be visible in the various software centers, but it may be harder to find.

Comments welcome.

Here’s a little Christmas present. This is GNOME Software working with xdg-app to allow live updates of apps and runtimes.

This is very much a prototype and needs a lot more work, but seems to work for me with xdg-app from git master (compile with --enable-libxdgapp). If any upstream projects needed any more encouragement, not including an AppData file means the application gets marked as nonfree as we don’t have any project licensing information. Inkscape, I’m looking at you.

I’m finally able to talk about one of the large vendors who have been trialing the LVFS service for the last few months. Dell have been uploading embargoed UEFI firmware files with metadata for a while, testing the process and the workflow ready for upcoming new models. Mario (Dell) and myself (Red Hat) have been working on fixing all the issues that pop up on real hardware and making the web service both secure and easy to use.

The Dell Edge Gateway will be available for purchase soon. When it goes on sale, firmware updates in Linux will work out-of-the-box. I’ve been told that Dell are considering expanding out the LVFS support to all new models supporting UEFI updates. In order to prioritize what models to work on first, I’ve been asked to share this anonymous survey on what Dell hardware people are using on Linux and to gauge if people actually care about being able to upgrade the firmware easily in Linux.

In November, 224 firmware files were installed onto client systems using fwupd. At the moment to update the firmware metadata you need to manually click the refresh button in the updates page, which so far 40,000 people have done. Given that the ColorHug hardware is the only released hardware with firmware on the LVFS, the 224 downloads is about what I expected. When we have major vendors like Dell (and other vendors I can’t talk about yet) shipping real consumer hardware with UEFI update capability the number of files provided should go up by orders of magnitude.

For Fedora 24 we’ll be downloading the firmware metadata automatically (rather than requiring a manual refresh in the updates panel) and we’ve been using the Fedora 23 users as a good way of optimizing the service so we know we can handle the load when we get hundreds of thousands of automatic requests a month. Fedora 24 will also be the first release able to do updates on DFU USB devices, and also the first release with system upgrade capabilities inside GNOME Software so it’s quite exciting from my point of view.

With Dell on board, I’m hoping it will give some of the other vendors enough confidence in the LVFS to talk about distributing their own firmware in public. The LVFS is something I run for all distributions free of charge, but of course Red Hat pays for my time to develop and run the service. I’m looking forward to working with more Red Hat partners and OpenHardware vendors adding even more firmware for even more types of device in the future.

I posted a few weeks ago about the difficulty of providing device-side verification of firmware updates, at the same time remaining OpenHardware and thus easily hackable. The general consensus was that allowing anyone to write any kind of firmware to the device without additional authentication was probably a bad idea, even for OpenHardware devices. I think I’ve come up with an acceptable compromise I can write up as a recommendation, as per usual using the ColorHug+ as an example. For some background, I’ve sold nearly 3,000 original ColorHug devices, and in the last 4 years just three people wanted help writing custom firmware, so I hope you can see the need to protect the majority is so much larger than making the power users happy.

ColorHug+ will be supplied with a bootloader that accepts only firmware encrypted with the secret XTEA key I that I’m using for my devices. XTEA is an acceptable compromise between something as secure as ECC, but that’s actually acceptable in speed and memory usage for a 8-bit microcontroller running at 6MHz with 8k of ROM. Flashing a DIY or modified firmware isn’t possible, and by the same logic flashing a malicious firmware will also not work.

To unlock the device (and so it stays OpenHardware) you just have to remove the two screws, and use a paper-clip to connect TP5 and GND while the device is being plugged into the USB port. Both lights will come on, and stay on for 5 seconds and then the code protection is turned off. This means you can now flash any home-made or malicious firmware to the device as you please.

There are downsides to unlocking; you can’t re-lock the hardware so it supports official updates again. I don’t know if this is a huge problem; flashing home-made firmware could damage the device (e.g. changing the pin mapping from input to output and causing something to get hot). If this is a huge problem I can fix CH+ to allow re-locking and fix up the guidelines, although I’m erring on unlocking being a one way operation.

Comments welcome.

For quite a long time fwupd has supported updating the system ‘BIOS’ using the UpdateCapsule UEFI mechanism. This open specification allows vendors provide a single update suitable for Windows and Linux, and the mechanism for applying it is basically the same for all vendors. Although there are only a few systems in the wild supporting capsule updates, a lot of vendors are planning new models next year, and a few of the major ones have been trialing the LVFS service for quite a while too. With capsule updates, fwupd and the LVFS we now have a compelling story for how to distribute and securely install system BIOS updates automatically.

It’s not such a rosy story for USB devices. In theory, everything should be using the DFU specification which has been endorsed by the USB consortium, but for a number of reasons quite a few vendors don’t use this. I’m guilty as charged for the ColorHug devices, as I didn’t know of the existance of DFU when designing the hardware. For ColorHug I just implemented a vendor-specific HID bootloader with a few custom commands as so many other vendors have done; it works well, but every vendor does things a slightly different way which needs having vendor specific update tools and fairly random firmware file formats.

With DFU, what’s supposed to happen is there are two modes for the device, a normal application runtime which is doing whatever the device is supposed to be doing, and another DFU mode which is really just an EEPROM programmer. By ‘detaching’ the application firmware using a special interface you can program the device and then return to normal operation.

So, what to do? For fwupd I want to ask vendors of removable hardware to implement DFU so that we don’t need to write code for each device type in fwupd. To make this a compelling prospect I’ve spent a good chunk of time of last week:

• Creating a GObjectIntrospectable and cancellable host-side library called libdfu
• Writing a reference GPLv3+ device-side implementation for a commonly used USB stack for PIC microcontrollers
• Writing the interface code in fwupd to support DFU files wrapped in .cab files for automatic deployment

At the moment libdfu supports reading and writing raw, DFU and DfuSe file types, and supports reading and writing to DFU 1.1 devices. I’ve not yet implemented writing to ST devices (a special protocol extension invented by ST Microsystems) although that’s only because I’m waiting for someone to lend me a device with a STM32F107 included (e.g. DSO Nano). I’ve hopefully made the code flexible enough to make this possible without breaking API, although the libdfu library is currently private to fwupd until it’s had some proper review. You can of course use the dependable dfu-util tool to flash firmware, but this wasn’t suitable for use inside fwupd for various reasons.

Putting my money where my mouth is, I’ve converted the (not-yet-released) ColorHug+ bootloader and firmware to use DFU; excluding all the time I spent writing the m-stack patch and the libdfu support in fwupd it only took a couple of hours to build and test. Thanks to Christoph Brill, I’ll soon be getting some more hardware (a Neo FreeRunner) to verify this new firmware update mechanism on a real device with multiple implemented DFU interfaces. If anyone else has any DFU-capable hardware (especially Arduino-style devices) I’d be glad of any donations.

Once all this new code has settled down I’m going to be re-emailing a lot of the vendors who were unwilling to write vendor-specific code in fwupd. I’m trying to make the barrier to automatic updates on Linux as low as possible.

Comments welcome.

A long time ago, GNOME software used to show star ratings as popularity next to the application using the fedora-tagger application. This wasn’t a good idea for several reasons:

• People can’t agree on a scale. Is an otherwise flawless application with one translation issue 5 stars or 4? Is a useful computational fluid dynamics application that crashes on startup but can be run manually on the command line 1 star or 3 stars?
• It only worked on Fedora, and there was no real policy on how to share data, or the privacy implications of clicking a star
• People could “game” the ratings system, for example hardcore KDE users could go through all the GNOME apps and give then one star. We then limited this to only rate applications that you have installed, but it was really a cat and mouse thing.

So, lets go two steps back. What is the star rating trying to convey to the user? When I look at a star rating, I want to see a proportional number of stars to how awesome it is to me. The rest of this blog tries to define awesomeness.

As part of the AppStream generation process we explode various parts of the distro binary package and try to build metadata by merging various sources together, for example AppData, desktop files and icons. As part of this we also have access to the finished binary and libraries, and so can also run tools on them to get a metric of awesomeness. So far, the metrics of awesomeness (here-on known as “kudos”) are:

• AppMenu — has an application menu in line with the GNOME 3 HIG
• HiDpiIcon — installs a 128×128 or larger application icon
• HighContrast — installs hicontrast icons for visually impaired users
• ModernToolkit — uses a modern toolkit like Gtk-3 or QT-5
• Notifications — registers desktop notifications
• SearchProvider — provides a search provider for GNOME Shell or KDE Plasma
• UserDocs — provides user documentation

These attempt to define how tightly the application is integrated with the platform, which is usually a pretty good metric of awesomeness. Of course, some applications like Blender are an island in terms of integration, but of course are awesome. We still need new ideas for this, so ideas are very much welcome.

There are some other “run-time” kudos used as well. These are not encoded by the builder as they require some user information or are too specific to GNOME Software. These include:

• FeaturedRecommended — One of the GNOME Software design team chose to feature this
• HasKeywords — there are keywords in the desktop file used for searching
• HasScreenshots — more than one screenshot is supplied
• MyLanguage — has a populated translation in my locale, or a locale fallback
• PerfectScreenshots — screenshots are perfectly sized, in 16:9 aspect
• Popular — lots of people have downloaded this (only available on Fedora)
• RecentRelease — there been an upstream release in the last year

When added together, the number of stars will correspond roughtly to the number of kudos the application has.

You can verify the kudos your application is getting by doing something like:

killall gnome-software
gnome-software --verbose

and then navigating to the details for an application you’ll see on the console:

 id-kind:         desktop
 state:           available
 id:              blender.desktop
 kudo:            recent-release
 kudo:            featured-recommended
 kudo:            has-screenshots
 kudo:            popular
 kudo-percentage: 60

Comments (as always) are welcome, as are new ideas on how to test for awesomeness.

This weekend I finished the penultimate feature for the LVFS. Before today, when uploading firmware there was up to a 24h delay before the new firmware would appear in the metadata. This was because there was a cronjob on my home server downloading files every night from the LVFS site, running appstream-builder on them locally and then uploading the metadata back to the site. Not awesome at all.

Actually generating the metadata in the OpenShift instance was impossible, until today. Due to libgcab and libappstream-glib not being available on the RHEL 6.2 instance I’m using, I had to re-implement two things in Python:

• Reading and writing Microsoft cabinet archives
• Reading MetaInfo files and writing compressed AppStream XML

The two helper libraries (only really implementing the parts required, but patches welcome) are python-cabarchive and python-appstream. I’m not awesome at Python, so feedback (in the form of pull requests) welcome.

This means I’m nearly okay to be hit by a bus. Well, nearly; the final feature is to collect statistics about how many people are downloading each firmware file, and possibly collecting data on how many failures and successes there have been when actually applying the firmware. This is actually quite tricky to do without causing privacy issues and not doing double counting. I’ll do some more thinking and then write up a proposal, ideas welcome.