Different User Types in LVFS

I’ve been working with two large (still un-named) vendors about their required features for the Linux Vendor Firmware Service. One of the new features I’ve landed this week in the test instance are the different user modes.

Screenshot from 2015-08-18 21-45-31

There are currently three classes of user:

  • The admin user that can do anything
  • Unprivileged users that can just upload files to the testing target
  • QA users that can upload files to the testing or stable target, and can tag files from testing to stable

This allows a firmware engineer to upload files before the new hardware has launched, and then someone else from the QA or management team can test the firmare and then push it out to the firmware so it can be flashed on real hardware by users.

I’ve also added functionality so that users can also change their own password (moving away from vendor keys) and added a simple test suite to test all the new rules.

Linux Vendor Firmware Service: We Need Your Help

I spend a lot of my day working on framework software for other programs to use. I enjoy this plumbing, and Red Hat gives me all the time I need to properly design and build these tricky infrastructure-type projects. Sometimes, just one person isn’t enough.

For the LVFS project, I need vendors making hardware to submit firmware files with carefully written metadata so that they can be downloaded by hundreds of thousands of Linux users securely and automatically. I also need those vendors to either use a standardized flashing protocol (e.g. DFU or UEFI) or to open the device specifications enough to allow flashing firmware without signing an NDA.

Over the last couple of months I’ve been emailing various tech companies trying to get hold of the right people to implement this. So far the reaction from companies has been enthusiastic and apathetic in equal measures. I’ve had a few vendors testing the process, but I can’t share those names just yet as most companies have been testing with unreleased hardware.

This is where you come in. On your Linux computer right now, think about what hardware you own that works in Linux that you know has user-flashable firmware? What about your BIOS, your mouse, or your USB3 hub? Your network card, your RAID card, or your video card?

Things I want you to do:

  • Find the vendor on the internet, and either raise a support case or send an email. Try and find a technical contact, not just some sales or marketing person
  • Tell the vendor that you would like firmware updates when using Linux, and that you’re not able to update the firmware booting to Windows or OS-X
  • Tell the vendor that you’re more likely to buy from them again if firmware updates work on Linux
  • Inform the vendor about the LVFS project : http://www.fwupd.org/

At all times I need you to be polite and courteous, after all we’re asking the vendor to spend time (money) on doing something extra for a small fraction of their userbase. Ignoring one email from me is easy, but getting tens or hundreds of support tickets about the same issue is a great way to get an issue escalated up to the people that can actually make changes.

So please, spend 15 minutes opening a support ticket or sending an email to a vendor now.

Building a better catalog file

Inside a windows driver package you’ll probably see a few .dll‘s, a .inf file and a .cat file. If you’ve ever been curious in Windows you’ll have double clicked it and it would show some technical information about the driver and some information about who signed the file.

We want to use this file to avoid having to get vendors to manually sign the firmware file with a GPG detached signature, which also implies trusting the Microsoft WHQL certificate. These are my notes on my adventure so far.

There are not many resources on this stuff, and I’d like to thank dwmw2 and dhowels for all their help so far answering all my stupid questions. osslsigncode is also useful to see how other signing is implemented.

So, the basics. A .cat file is a SMIME PKCS DER file. We can dump the file using:

openssl asn1parse -in ecfirmware.cat  -inform DER

and if we were signing just one file we should be able to verify the .cat file with something like this:

wget http://www.microsoft.com/pki/certs/MicRooCerAut_2010-06-23.crt
openssl x509 -in MicRooCerAut_2010-06-23.crt -inform DER -out ms/msroot.pem -outform PEM
cat ms/*.pem > ms/certs.pem
openssl smime -verify -CAfile ms/certs.pem -in ecfirmware.cat -inform DER -attime $(date +%s --date=2015-01-01) -content ECFirmware.
Verification failed

(Ignore the need to have the root certificate for now, that seems to be a bug in OpenSSL and they probably have bigger fires to fight at this point)

…but it’s not. We have a pkcs7-signed blob and we need to work out how to get the signature to actually *match* and then we have to work out how to interpret the pkcs7-signed data blob, and use the sha256sums therein to validate the actual data. OpenSSL doesn’t know how to interpret the MS content type OID ( so it wimps out and doesn’t put any data into the digest at all.

We can get the blob using a simple:

dd if=ecfirmware.cat of=ecfirmware.cat.payload bs=1 skip=66 count=1340

…which now verifies:

openssl smime -verify -CAfile ms/certs.pem -in ecfirmware.cat -inform DER -attime $(date +%s --date=2015-01-01) -content ecfirmware.cat.payload
Verification successful

The blob appears to be a few tables of UTF-16 filename and SHA1/SHA256 checksummed data, encoded in ASN.1 notation. I’ve spent quite a few evenings decompiling the DER file into an ASN file without a whole lot of success (there are 14(!) layers of structures to contend with) and I’ve still not got an ASN file that can correctly parse my DER file for my simple unsigned v1 (ohh yes, v1 = SHA1, v2 = SHA256) test files. There is also a lot of junk data in the blob, and some questionable design choices which mean it’s a huge pain to read. Even if I manage to write the code to read the .cat data blob I’ve then got to populate the data (including the junk data…) so that Windows will accept my file to avoid needing a Microsoft box to generate all firmware images. Also add to the mix that the ASN.1 data is different on different Windows versions (with legacy versions overridden), which explains why you see things like rather than translated titles in the catalog viewer in Windows XP when trying to view .cat files created on Windows 7.

I’ve come to the conclusion that writing a library to reliably read and write all versions of .cat files is probably about 3 months work, and that’s 3 months I simply don’t have. Given there isn’t actually a specification (apart from a super small guide on how to use the MS crypto API) it would also be an uphill battle with every Windows release.

We could of course do something Linux specific that does the same thing, although that obviously would not work on Windows and means we have to ask the vendors to do an extra step in release engineering. Using GPG would be easiest, but a lot of the hardware vendors seem wed to the PKCS certificate mechanism, and I suppose it does mean you can layer certificates for root trust, vendors and contractors. GPG signing the firmware file only doesn’t actually give us a file-list with the digest values of the other metadata in the .cab file.

A naive solution would be to do something like this:

sha25sum firmware.inf firmware.metainfo.xml firmware.bin > firmware.digest
openssl dgst -sha256 -sign cert-private.pem -out firmware.sign firmware.digest
openssl dgst -sha256 -verify cert-pubkey.pem -signature firmware.sign firmware.files

But to actually extract the firmware.digest file we need the private key. We can check prepared data using the public key, but that means shipping firmware.digest and firmware.sign when we only really want one file (.cab files checksum the files internally, so we can be sure against data corruption).

Before I go crazy and invent yet another file format specification does anybody know of a signed digest format with an open specification? Better ideas certainly welcome, thanks.


Parsing Option ROM Firmware

A few weeks ago an issue was opened on fwupd by pippin. He was basically asking for a command to return all the hashes of the firmwares installed on his hardware, which I initially didn’t really see the point of doing. However, after doing a few hours research about all the malware that can hide in VBIOS for graphics cards, option ROM in network cards, and keyboard matrix EC processors I was suitably worried also. I figured fixing the issue was a good idea. Of course, malware could perhaps hide itself (i.e. hiding in an unused padding segment and masking itself out on read) but this at least raises the bar from a security audit point of view, and is somewhat easier than opening the case and attaching a SPI programmer to the chip itself.

Fast forward a few nights. We can now verify ATI, NVIDIA, INTEL and ColorHug firmware. I’ve not got any other hardware with ROM that I can read from userspace, so this is where I need your help. I need willing volunteers to compile fwupd from git master (or rebuild my srpm) and then run:

cd fwupd/src
find /sys/devices -name rom -exec sudo ./fwupdmgr dump-rom {} \;

All being well you should see something like this:

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom -> f21e1d2c969dedbefcf5acfdab4fa0c5ff111a57 [Version:]

If you see something just that, you’re not super helpful to me. If you see Error reading from file: Input/output error then you’re also not so helpful as the kernel module for your hardware is exporting a rom file and not hooking up the read vfuncs. If you get an error like Failed to detect firmware header [8950] or Firmware version extractor not known then you’ve just become interesting. If that’s you, can you send the rom file to richard_at_hughsie.com as an attachment along with any details you know about the hardware. Thanks!


Introducing the Linux Vendor Firmware Service

As some of you may know, I’ve spent the last couple of months talking with various Red Hat partners and other OpenHardware vendors that produce firmware updates. These include most of the laptop vendors that you know and love, along with a few more companies making very specialized hardware.

We’ve now got a process, fwupd, that is capable of taking the packaged update and applying it to the hardware using various forms of upload mechanism. We’ve got a specification, AppStream, which is used to describe the updates and provide metadata for what firmware updates are available to be installed. What we were missing was to “close the circle” and provide a web service for small and medium size vendors to use to upload new firmware and make it available to Linux users.

Microsoft already provides such a thing for vendors to use, and it’s part of the Microsoft Update service. From the vendors I’ve talked to, the majority don’t want to run any tools on their firmware to generate metadata. Most of them don’t even want to commit to hosting the metadata or firmware files in the same place forever, and with a couple of exceptions actually like the Microsoft Update model.

I’ve created a simple web service that’s being called Linux Vendor Firmware Service (perhaps not the final name). You can see the site in action here, although it’s not terribly useful or exciting if you’re not a hardware vendor.

If you are vendor that produces firmware and want an access key for the beta site, please let me know. All firmware uploaded will be transferred to the final site, although I’m still waiting to hear back from Red Hat legal about a longer version of the redistribution agreement.

Anyway, comments very welcome, thanks.

ColorHugALS and Sensor HID

As Bastien hinted in his last blog post, we now have some new test firmware for the ColorHugALS device. The ever-awesome Benjamin Tissoires has been hacking on an alternative device firmware, this time implementing the Sensor HID interface that Microsoft is suggesting vendors use for internal ambient light sensors on tablets and laptops for Windows 8.

Implementing this new interface has several advantages:

  • The sensor should “just work” with Windows 8 without a driver
  • The sensor now works with iio-sensor-proxy without writing any interface code
  • We can test the HID code in the kernel with a device we can hack to do strange things
  • colorhug-als1-large

    So, if you want to test the new GNOME ambient light sensor code, flash your ColorHugALS with this file using colorhug-cmd flash-firmware ColorHugALS-SensorHID.bin — the flash process will appear to fail right at the end, but this is just because we’ve not yet written the HID version of the SetFlashSuccess call that instructs the bootloader to start the firmware automatically when inserted. This isn’t actually such a bad thing for an experimental firmware, but means when you remove then insert your ALS device you’ll have to do colorhug-cmd boot-flash to switch from the flashing red LED bootloader mode into the new firmware mode.

    If it’s too broken for you right now, you can go back to the real firmware using colorhug-cmd when in bootloader mode.


    There are still 17 ColorHugALS devices in stock, if you want to buy one for testing. Once they’ve gone, they’re gone, I don’t think I’ll be building another batch unless there’s a lot more demand as right now I’m building them at a loss.

    Fedora 22 and missing applications

    Quite a few people are going to be installing Fedora 22 in the coming days, searching for things in the software center and not finding what they want. This is because some applications still don’t ship AppData files, which have become compulsory for this release. So far, over 53% of applications shipped in Fedora ship the required software center metadata, up from the original 12% in Fedora 21. If you don’t like this, you can either use dnf to install the package on the command line, or set gsettings set org.gnome.software require-appdata false. If you want to see your application in the software center in the future, please file a bug either upstream or downstream (I’ve already filed a lot of upstream bugs) or even better write the metadata and get it installed either upstream tarball or downstream in the Fedora package. Most upstream and downstream maintainers have shipped the extra software center information, but some others might need a little reminder about it from users.

    Telling the user about firmware updates

    A common use-case that has appeared over the last week is that some vendors just want to notify people there is updated firmware available, but they don’t want fwupd to apply it automatically. This might be because an external programmer is required to update, the flashing tool is non-free, or other manual steps are required.

    If anyone is interested in doing this for their device, there are just two USB string descriptors to add, and then it all just magically works once AppStream metadata is supplied. The device doesn’t have to be OpenHardware, so there’s no real excuse.

    Updating OpenHardware Firmware 2

    After quite a bit of peer review, it turns out my idea to use the unused serial field wasn’t so awesome. Thanks mostly to Stefan, we have a new proposal.

    For generic USB devices you can use a firmware version extension that is now used by ColorHug, and I hope other projects too in the future. With this the fwupd daemon can obtain the firmware version without claiming the interface on the device and preventing other software from using it straight away.

    To implement the firmware version extension just create an interface descriptor with class code 0xff, subclass code 0x46 and protocol 0x57 pointing to a string descriptor with the firmware version.
    An example commit to the ColorHug2 firmware can be found here. It costs just 21 bytes of ROM which means it’s suitable even for resource-constrained devices like the ColorHugALS. Comments welcome.

    Updating OpenHardware Firmware

    EDIT: Don’t implement this. See the follow-up post.

    One of the use-cases I’ve got for fwupd is for updating firmware on small OpenHardware projects. It doesn’t make sense for each of the projects to write a GUI firmware flash program when most of them are using a simple HID or DFU bootloader to do basically the same thing. We can abstract out the details, and just require the upstream project to provide metadata about what is fixed in each update that we can all share.

    The devil, as they say, is in the details. When enumerating devices, fwupd needs to know the device GUID (usually, just a hardcoded mapping table from USB VID/PID). This certainly could be in a udev rule that can be dropped into the right place when developing a new device, as I don’t want people to have to build a fwupd from git just to update the new shiny device that’s just arrived.

    There are two other things fwupd needs to know. The most important is the current firmware version for a device. There is no specification for this as far as I can tell. ColorHug has a HID command GET_VERSION which returns 3 uint16M numbers for the major, minor and micro versions and other device firmwares have other similarly obvious but different ways of doing it.

    The other is how to switch the device in firmware mode back into bootloader mode so that it can flash a new version. For ColorHug there’s a RESET command, but on other hardware it’s either a custom command sequence, or doing something physical like pressing a secret button with a paperclip or shorting two pins on a PCB.

    I think it would be useful to notify the user that there in an update available, even if we can’t actually do the upgrade without doing some manual step. For this we need to get the current firmware version, ideally without open()ing the device as this will prevent other software from using it straight away. What we can get from the device for free is the device descriptors.

    What I’m going to do for ColorHug is to change the unused device serial string descriptor to “FW:1.2.3”. I’ll also support in fwupd devices changing the product string from “Widget” to “Widget FW:1.2.3” i.e. we look in the various strings for a token with a “FW:” prefix and use that.

    If that isn’t specified then we can fall back to opening the device and doing a custom command, but when you can ask friendly upstream firmware vendors to make a super small change, it makes things much easier for everyone. Comments welcome.