Technical Blog of Richard Hughes

I posted a few weeks ago about the difficulty of providing device-side verification of firmware updates, at the same time remaining OpenHardware and thus easily hackable. The general consensus was that allowing anyone to write any kind of firmware to the device without additional authentication was probably a bad idea, even for OpenHardware devices. I think I’ve come up with an acceptable compromise I can write up as a recommendation, as per usual using the ColorHug+ as an example. For some background, I’ve sold nearly 3,000 original ColorHug devices, and in the last 4 years just three people wanted help writing custom firmware, so I hope you can see the need to protect the majority is so much larger than making the power users happy.

ColorHug+ will be supplied with a bootloader that accepts only firmware encrypted with the secret XTEA key I that I’m using for my devices. XTEA is an acceptable compromise between something as secure as ECC, but that’s actually acceptable in speed and memory usage for a 8-bit microcontroller running at 6MHz with 8k of ROM. Flashing a DIY or modified firmware isn’t possible, and by the same logic flashing a malicious firmware will also not work.

To unlock the device (and so it stays OpenHardware) you just have to remove the two screws, and use a paper-clip to connect TP5 and GND while the device is being plugged into the USB port. Both lights will come on, and stay on for 5 seconds and then the code protection is turned off. This means you can now flash any home-made or malicious firmware to the device as you please.

There are downsides to unlocking; you can’t re-lock the hardware so it supports official updates again. I don’t know if this is a huge problem; flashing home-made firmware could damage the device (e.g. changing the pin mapping from input to output and causing something to get hot). If this is a huge problem I can fix CH+ to allow re-locking and fix up the guidelines, although I’m erring on unlocking being a one way operation.

Comments welcome.

For quite a long time fwupd has supported updating the system ‘BIOS’ using the UpdateCapsule UEFI mechanism. This open specification allows vendors provide a single update suitable for Windows and Linux, and the mechanism for applying it is basically the same for all vendors. Although there are only a few systems in the wild supporting capsule updates, a lot of vendors are planning new models next year, and a few of the major ones have been trialing the LVFS service for quite a while too. With capsule updates, fwupd and the LVFS we now have a compelling story for how to distribute and securely install system BIOS updates automatically.

It’s not such a rosy story for USB devices. In theory, everything should be using the DFU specification which has been endorsed by the USB consortium, but for a number of reasons quite a few vendors don’t use this. I’m guilty as charged for the ColorHug devices, as I didn’t know of the existance of DFU when designing the hardware. For ColorHug I just implemented a vendor-specific HID bootloader with a few custom commands as so many other vendors have done; it works well, but every vendor does things a slightly different way which needs having vendor specific update tools and fairly random firmware file formats.

With DFU, what’s supposed to happen is there are two modes for the device, a normal application runtime which is doing whatever the device is supposed to be doing, and another DFU mode which is really just an EEPROM programmer. By ‘detaching’ the application firmware using a special interface you can program the device and then return to normal operation.

So, what to do? For fwupd I want to ask vendors of removable hardware to implement DFU so that we don’t need to write code for each device type in fwupd. To make this a compelling prospect I’ve spent a good chunk of time of last week:

• Creating a GObjectIntrospectable and cancellable host-side library called libdfu
• Writing a reference GPLv3+ device-side implementation for a commonly used USB stack for PIC microcontrollers
• Writing the interface code in fwupd to support DFU files wrapped in .cab files for automatic deployment

At the moment libdfu supports reading and writing raw, DFU and DfuSe file types, and supports reading and writing to DFU 1.1 devices. I’ve not yet implemented writing to ST devices (a special protocol extension invented by ST Microsystems) although that’s only because I’m waiting for someone to lend me a device with a STM32F107 included (e.g. DSO Nano). I’ve hopefully made the code flexible enough to make this possible without breaking API, although the libdfu library is currently private to fwupd until it’s had some proper review. You can of course use the dependable dfu-util tool to flash firmware, but this wasn’t suitable for use inside fwupd for various reasons.

Putting my money where my mouth is, I’ve converted the (not-yet-released) ColorHug+ bootloader and firmware to use DFU; excluding all the time I spent writing the m-stack patch and the libdfu support in fwupd it only took a couple of hours to build and test. Thanks to Christoph Brill, I’ll soon be getting some more hardware (a Neo FreeRunner) to verify this new firmware update mechanism on a real device with multiple implemented DFU interfaces. If anyone else has any DFU-capable hardware (especially Arduino-style devices) I’d be glad of any donations.

Once all this new code has settled down I’m going to be re-emailing a lot of the vendors who were unwilling to write vendor-specific code in fwupd. I’m trying to make the barrier to automatic updates on Linux as low as possible.

Comments welcome.

A long time ago, GNOME software used to show star ratings as popularity next to the application using the fedora-tagger application. This wasn’t a good idea for several reasons:

• People can’t agree on a scale. Is an otherwise flawless application with one translation issue 5 stars or 4? Is a useful computational fluid dynamics application that crashes on startup but can be run manually on the command line 1 star or 3 stars?
• It only worked on Fedora, and there was no real policy on how to share data, or the privacy implications of clicking a star
• People could “game” the ratings system, for example hardcore KDE users could go through all the GNOME apps and give then one star. We then limited this to only rate applications that you have installed, but it was really a cat and mouse thing.

So, lets go two steps back. What is the star rating trying to convey to the user? When I look at a star rating, I want to see a proportional number of stars to how awesome it is to me. The rest of this blog tries to define awesomeness.

As part of the AppStream generation process we explode various parts of the distro binary package and try to build metadata by merging various sources together, for example AppData, desktop files and icons. As part of this we also have access to the finished binary and libraries, and so can also run tools on them to get a metric of awesomeness. So far, the metrics of awesomeness (here-on known as “kudos”) are:

• AppMenu — has an application menu in line with the GNOME 3 HIG
• HiDpiIcon — installs a 128×128 or larger application icon
• HighContrast — installs hicontrast icons for visually impaired users
• ModernToolkit — uses a modern toolkit like Gtk-3 or QT-5
• Notifications — registers desktop notifications
• SearchProvider — provides a search provider for GNOME Shell or KDE Plasma
• UserDocs — provides user documentation

These attempt to define how tightly the application is integrated with the platform, which is usually a pretty good metric of awesomeness. Of course, some applications like Blender are an island in terms of integration, but of course are awesome. We still need new ideas for this, so ideas are very much welcome.

There are some other “run-time” kudos used as well. These are not encoded by the builder as they require some user information or are too specific to GNOME Software. These include:

• FeaturedRecommended — One of the GNOME Software design team chose to feature this
• HasKeywords — there are keywords in the desktop file used for searching
• HasScreenshots — more than one screenshot is supplied
• MyLanguage — has a populated translation in my locale, or a locale fallback
• PerfectScreenshots — screenshots are perfectly sized, in 16:9 aspect
• Popular — lots of people have downloaded this (only available on Fedora)
• RecentRelease — there been an upstream release in the last year

When added together, the number of stars will correspond roughtly to the number of kudos the application has.

You can verify the kudos your application is getting by doing something like:

killall gnome-software
gnome-software --verbose

and then navigating to the details for an application you’ll see on the console:

 id-kind:         desktop
 state:           available
 id:              blender.desktop
 kudo:            recent-release
 kudo:            featured-recommended
 kudo:            has-screenshots
 kudo:            popular
 kudo-percentage: 60

Comments (as always) are welcome, as are new ideas on how to test for awesomeness.

This weekend I finished the penultimate feature for the LVFS. Before today, when uploading firmware there was up to a 24h delay before the new firmware would appear in the metadata. This was because there was a cronjob on my home server downloading files every night from the LVFS site, running appstream-builder on them locally and then uploading the metadata back to the site. Not awesome at all.

Actually generating the metadata in the OpenShift instance was impossible, until today. Due to libgcab and libappstream-glib not being available on the RHEL 6.2 instance I’m using, I had to re-implement two things in Python:

• Reading and writing Microsoft cabinet archives
• Reading MetaInfo files and writing compressed AppStream XML

The two helper libraries (only really implementing the parts required, but patches welcome) are python-cabarchive and python-appstream. I’m not awesome at Python, so feedback (in the form of pull requests) welcome.

This means I’m nearly okay to be hit by a bus. Well, nearly; the final feature is to collect statistics about how many people are downloading each firmware file, and possibly collecting data on how many failures and successes there have been when actually applying the firmware. This is actually quite tricky to do without causing privacy issues and not doing double counting. I’ll do some more thinking and then write up a proposal, ideas welcome.

For the last couple of days I’ve been working with a large vendor adding new functionality to the LVFS to support their specific workflow.

The new embargo target allows vendors to test the automatic update functionality using a secret vendor-specific URL set in /etc/fwupd.conf without releasing it to the general public until the hardware has been announced.

Updates normally go through these stages: Private → Embargoed → Testing → Stable although LVFS users with the QA capability can skip these as required. The screenshot also shows that we’re unpacking the .cab file and parsing the metainfo file server side (in python), which gives us so much more rich detail about the firmware.

I’ve been working with two large (still un-named) vendors about their required features for the Linux Vendor Firmware Service. One of the new features I’ve landed this week in the test instance are the different user modes.

There are currently three classes of user:

• The admin user that can do anything
• Unprivileged users that can just upload files to the testing target
• QA users that can upload files to the testing or stable target, and can tag files from testing to stable

This allows a firmware engineer to upload files before the new hardware has launched, and then someone else from the QA or management team can test the firmare and then push it out to the firmware so it can be flashed on real hardware by users.

I’ve also added functionality so that users can also change their own password (moving away from vendor keys) and added a simple test suite to test all the new rules.

I spend a lot of my day working on framework software for other programs to use. I enjoy this plumbing, and Red Hat gives me all the time I need to properly design and build these tricky infrastructure-type projects. Sometimes, just one person isn’t enough.

For the LVFS project, I need vendors making hardware to submit firmware files with carefully written metadata so that they can be downloaded by hundreds of thousands of Linux users securely and automatically. I also need those vendors to either use a standardized flashing protocol (e.g. DFU or UEFI) or to open the device specifications enough to allow flashing firmware without signing an NDA.

Over the last couple of months I’ve been emailing various tech companies trying to get hold of the right people to implement this. So far the reaction from companies has been enthusiastic and apathetic in equal measures. I’ve had a few vendors testing the process, but I can’t share those names just yet as most companies have been testing with unreleased hardware.

This is where you come in. On your Linux computer right now, think about what hardware you own that works in Linux that you know has user-flashable firmware? What about your BIOS, your mouse, or your USB3 hub? Your network card, your RAID card, or your video card?

Things I want you to do:

• Find the vendor on the internet, and either raise a support case or send an email. Try and find a technical contact, not just some sales or marketing person
• Tell the vendor that you would like firmware updates when using Linux, and that you’re not able to update the firmware booting to Windows or OS-X
• Tell the vendor that you’re more likely to buy from them again if firmware updates work on Linux
• Inform the vendor about the LVFS project : http://www.fwupd.org/

At all times I need you to be polite and courteous, after all we’re asking the vendor to spend time (money) on doing something extra for a small fraction of their userbase. Ignoring one email from me is easy, but getting tens or hundreds of support tickets about the same issue is a great way to get an issue escalated up to the people that can actually make changes.

So please, spend 15 minutes opening a support ticket or sending an email to a vendor now.

Inside a windows driver package you’ll probably see a few .dll‘s, a .inf file and a .cat file. If you’ve ever been curious in Windows you’ll have double clicked it and it would show some technical information about the driver and some information about who signed the file.

We want to use this file to avoid having to get vendors to manually sign the firmware file with a GPG detached signature, which also implies trusting the Microsoft WHQL certificate. These are my notes on my adventure so far.

There are not many resources on this stuff, and I’d like to thank dwmw2 and dhowels for all their help so far answering all my stupid questions. osslsigncode is also useful to see how other signing is implemented.

So, the basics. A .cat file is a SMIME PKCS DER file. We can dump the file using:

openssl asn1parse -in ecfirmware.cat  -inform DER

and if we were signing just one file we should be able to verify the .cat file with something like this:

wget http://www.microsoft.com/pki/certs/MicRooCerAut_2010-06-23.crt
openssl x509 -in MicRooCerAut_2010-06-23.crt -inform DER -out ms/msroot.pem -outform PEM
cat ms/*.pem > ms/certs.pem
openssl smime -verify -CAfile ms/certs.pem -in ecfirmware.cat -inform DER -attime $(date +%s --date=2015-01-01) -content ECFirmware.38.7.50.0.cap
Verification failed

(Ignore the need to have the root certificate for now, that seems to be a bug in OpenSSL and they probably have bigger fires to fight at this point)

…but it’s not. We have a pkcs7-signed blob and we need to work out how to get the signature to actually *match* and then we have to work out how to interpret the pkcs7-signed data blob, and use the sha256sums therein to validate the actual data. OpenSSL doesn’t know how to interpret the MS content type OID (1.3.6.1.4.1.311.10.1) so it wimps out and doesn’t put any data into the digest at all.

We can get the blob using a simple:

dd if=ecfirmware.cat of=ecfirmware.cat.payload bs=1 skip=66 count=1340

…which now verifies:

openssl smime -verify -CAfile ms/certs.pem -in ecfirmware.cat -inform DER -attime $(date +%s --date=2015-01-01) -content ecfirmware.cat.payload
Verification successful

The blob appears to be a few tables of UTF-16 filename and SHA1/SHA256 checksummed data, encoded in ASN.1 notation. I’ve spent quite a few evenings decompiling the DER file into an ASN file without a whole lot of success (there are 14(!) layers of structures to contend with) and I’ve still not got an ASN file that can correctly parse my DER file for my simple unsigned v1 (ohh yes, v1 = SHA1, v2 = SHA256) test files. There is also a lot of junk data in the blob, and some questionable design choices which mean it’s a huge pain to read. Even if I manage to write the code to read the .cat data blob I’ve then got to populate the data (including the junk data…) so that Windows will accept my file to avoid needing a Microsoft box to generate all firmware images. Also add to the mix that the ASN.1 data is different on different Windows versions (with legacy versions overridden), which explains why you see things like 1.3.6.1.4.1.311.12.2.2 rather than translated titles in the catalog viewer in Windows XP when trying to view .cat files created on Windows 7.

I’ve come to the conclusion that writing a library to reliably read and write all versions of .cat files is probably about 3 months work, and that’s 3 months I simply don’t have. Given there isn’t actually a specification (apart from a super small guide on how to use the MS crypto API) it would also be an uphill battle with every Windows release.

We could of course do something Linux specific that does the same thing, although that obviously would not work on Windows and means we have to ask the vendors to do an extra step in release engineering. Using GPG would be easiest, but a lot of the hardware vendors seem wed to the PKCS certificate mechanism, and I suppose it does mean you can layer certificates for root trust, vendors and contractors. GPG signing the firmware file only doesn’t actually give us a file-list with the digest values of the other metadata in the .cab file.

A naive solution would be to do something like this:

sha25sum firmware.inf firmware.metainfo.xml firmware.bin > firmware.digest
openssl dgst -sha256 -sign cert-private.pem -out firmware.sign firmware.digest
openssl dgst -sha256 -verify cert-pubkey.pem -signature firmware.sign firmware.files

But to actually extract the firmware.digest file we need the private key. We can check prepared data using the public key, but that means shipping firmware.digest and firmware.sign when we only really want one file (.cab files checksum the files internally, so we can be sure against data corruption).

Before I go crazy and invent yet another file format specification does anybody know of a signed digest format with an open specification? Better ideas certainly welcome, thanks.

Richard.

A few weeks ago an issue was opened on fwupd by pippin. He was basically asking for a command to return all the hashes of the firmwares installed on his hardware, which I initially didn’t really see the point of doing. However, after doing a few hours research about all the malware that can hide in VBIOS for graphics cards, option ROM in network cards, and keyboard matrix EC processors I was suitably worried also. I figured fixing the issue was a good idea. Of course, malware could perhaps hide itself (i.e. hiding in an unused padding segment and masking itself out on read) but this at least raises the bar from a security audit point of view, and is somewhat easier than opening the case and attaching a SPI programmer to the chip itself.

Fast forward a few nights. We can now verify ATI, NVIDIA, INTEL and ColorHug firmware. I’ve not got any other hardware with ROM that I can read from userspace, so this is where I need your help. I need willing volunteers to compile fwupd from git master (or rebuild my srpm) and then run:

cd fwupd/src
find /sys/devices -name rom -exec sudo ./fwupdmgr dump-rom {} \;

All being well you should see something like this:

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom -> f21e1d2c969dedbefcf5acfdab4fa0c5ff111a57 [Version: 013.012.000.019.000000]

If you see something just that, you’re not super helpful to me. If you see Error reading from file: Input/output error then you’re also not so helpful as the kernel module for your hardware is exporting a rom file and not hooking up the read vfuncs. If you get an error like Failed to detect firmware header [8950] or Firmware version extractor not known then you’ve just become interesting. If that’s you, can you send the rom file to richard_at_hughsie.com as an attachment along with any details you know about the hardware. Thanks!

Richard.

As some of you may know, I’ve spent the last couple of months talking with various Red Hat partners and other OpenHardware vendors that produce firmware updates. These include most of the laptop vendors that you know and love, along with a few more companies making very specialized hardware.

We’ve now got a process, fwupd, that is capable of taking the packaged update and applying it to the hardware using various forms of upload mechanism. We’ve got a specification, AppStream, which is used to describe the updates and provide metadata for what firmware updates are available to be installed. What we were missing was to “close the circle” and provide a web service for small and medium size vendors to use to upload new firmware and make it available to Linux users.

Microsoft already provides such a thing for vendors to use, and it’s part of the Microsoft Update service. From the vendors I’ve talked to, the majority don’t want to run any tools on their firmware to generate metadata. Most of them don’t even want to commit to hosting the metadata or firmware files in the same place forever, and with a couple of exceptions actually like the Microsoft Update model.

I’ve created a simple web service that’s being called Linux Vendor Firmware Service (perhaps not the final name). You can see the site in action here, although it’s not terribly useful or exciting if you’re not a hardware vendor.

If you are vendor that produces firmware and want an access key for the beta site, please let me know. All firmware uploaded will be transferred to the final site, although I’m still waiting to hear back from Red Hat legal about a longer version of the redistribution agreement.

Anyway, comments very welcome, thanks.