Technical Blog of Richard Hughes

A few weeks ago an issue was opened on fwupd by pippin. He was basically asking for a command to return all the hashes of the firmwares installed on his hardware, which I initially didn’t really see the point of doing. However, after doing a few hours research about all the malware that can hide in VBIOS for graphics cards, option ROM in network cards, and keyboard matrix EC processors I was suitably worried also. I figured fixing the issue was a good idea. Of course, malware could perhaps hide itself (i.e. hiding in an unused padding segment and masking itself out on read) but this at least raises the bar from a security audit point of view, and is somewhat easier than opening the case and attaching a SPI programmer to the chip itself.

Fast forward a few nights. We can now verify ATI, NVIDIA, INTEL and ColorHug firmware. I’ve not got any other hardware with ROM that I can read from userspace, so this is where I need your help. I need willing volunteers to compile fwupd from git master (or rebuild my srpm) and then run:

cd fwupd/src
find /sys/devices -name rom -exec sudo ./fwupdmgr dump-rom {} \;

All being well you should see something like this:

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom -> f21e1d2c969dedbefcf5acfdab4fa0c5ff111a57 [Version: 013.012.000.019.000000]

If you see something just that, you’re not super helpful to me. If you see Error reading from file: Input/output error then you’re also not so helpful as the kernel module for your hardware is exporting a rom file and not hooking up the read vfuncs. If you get an error like Failed to detect firmware header [8950] or Firmware version extractor not known then you’ve just become interesting. If that’s you, can you send the rom file to richard_at_hughsie.com as an attachment along with any details you know about the hardware. Thanks!

Richard.

As some of you may know, I’ve spent the last couple of months talking with various Red Hat partners and other OpenHardware vendors that produce firmware updates. These include most of the laptop vendors that you know and love, along with a few more companies making very specialized hardware.

We’ve now got a process, fwupd, that is capable of taking the packaged update and applying it to the hardware using various forms of upload mechanism. We’ve got a specification, AppStream, which is used to describe the updates and provide metadata for what firmware updates are available to be installed. What we were missing was to “close the circle” and provide a web service for small and medium size vendors to use to upload new firmware and make it available to Linux users.

Microsoft already provides such a thing for vendors to use, and it’s part of the Microsoft Update service. From the vendors I’ve talked to, the majority don’t want to run any tools on their firmware to generate metadata. Most of them don’t even want to commit to hosting the metadata or firmware files in the same place forever, and with a couple of exceptions actually like the Microsoft Update model.

I’ve created a simple web service that’s being called Linux Vendor Firmware Service (perhaps not the final name). You can see the site in action here, although it’s not terribly useful or exciting if you’re not a hardware vendor.

If you are vendor that produces firmware and want an access key for the beta site, please let me know. All firmware uploaded will be transferred to the final site, although I’m still waiting to hear back from Red Hat legal about a longer version of the redistribution agreement.

Anyway, comments very welcome, thanks.

As Bastien hinted in his last blog post, we now have some new test firmware for the ColorHugALS device. The ever-awesome Benjamin Tissoires has been hacking on an alternative device firmware, this time implementing the Sensor HID interface that Microsoft is suggesting vendors use for internal ambient light sensors on tablets and laptops for Windows 8.

Implementing this new interface has several advantages:

The sensor should “just work” with Windows 8 without a driver

The sensor now works with iio-sensor-proxy without writing any interface code

We can test the HID code in the kernel with a device we can hack to do strange things

So, if you want to test the new GNOME ambient light sensor code, flash your ColorHugALS with this file using colorhug-cmd flash-firmware ColorHugALS-SensorHID.bin — the flash process will appear to fail right at the end, but this is just because we’ve not yet written the HID version of the SetFlashSuccess call that instructs the bootloader to start the firmware automatically when inserted. This isn’t actually such a bad thing for an experimental firmware, but means when you remove then insert your ALS device you’ll have to do colorhug-cmd boot-flash to switch from the flashing red LED bootloader mode into the new firmware mode.

If it’s too broken for you right now, you can go back to the real firmware using colorhug-cmd when in bootloader mode.

There are still 17 ColorHugALS devices in stock, if you want to buy one for testing. Once they’ve gone, they’re gone, I don’t think I’ll be building another batch unless there’s a lot more demand as right now I’m building them at a loss.

Quite a few people are going to be installing Fedora 22 in the coming days, searching for things in the software center and not finding what they want. This is because some applications still don’t ship AppData files, which have become compulsory for this release. So far, over 53% of applications shipped in Fedora ship the required software center metadata, up from the original 12% in Fedora 21. If you don’t like this, you can either use dnf to install the package on the command line, or set gsettings set org.gnome.software require-appdata false. If you want to see your application in the software center in the future, please file a bug either upstream or downstream (I’ve already filed a lot of upstream bugs) or even better write the metadata and get it installed either upstream tarball or downstream in the Fedora package. Most upstream and downstream maintainers have shipped the extra software center information, but some others might need a little reminder about it from users.

A common use-case that has appeared over the last week is that some vendors just want to notify people there is updated firmware available, but they don’t want fwupd to apply it automatically. This might be because an external programmer is required to update, the flashing tool is non-free, or other manual steps are required.

If anyone is interested in doing this for their device, there are just two USB string descriptors to add, and then it all just magically works once AppStream metadata is supplied. The device doesn’t have to be OpenHardware, so there’s no real excuse.

After quite a bit of peer review, it turns out my idea to use the unused serial field wasn’t so awesome. Thanks mostly to Stefan, we have a new proposal.

For generic USB devices you can use a firmware version extension that is now used by ColorHug, and I hope other projects too in the future. With this the fwupd daemon can obtain the firmware version without claiming the interface on the device and preventing other software from using it straight away.

To implement the firmware version extension just create an interface descriptor with class code 0xff, subclass code 0x46 and protocol 0x57 pointing to a string descriptor with the firmware version.
An example commit to the ColorHug2 firmware can be found here. It costs just 21 bytes of ROM which means it’s suitable even for resource-constrained devices like the ColorHugALS. Comments welcome.

EDIT: Don’t implement this. See the follow-up post.

One of the use-cases I’ve got for fwupd is for updating firmware on small OpenHardware projects. It doesn’t make sense for each of the projects to write a GUI firmware flash program when most of them are using a simple HID or DFU bootloader to do basically the same thing. We can abstract out the details, and just require the upstream project to provide metadata about what is fixed in each update that we can all share.

The devil, as they say, is in the details. When enumerating devices, fwupd needs to know the device GUID (usually, just a hardcoded mapping table from USB VID/PID). This certainly could be in a udev rule that can be dropped into the right place when developing a new device, as I don’t want people to have to build a fwupd from git just to update the new shiny device that’s just arrived.

There are two other things fwupd needs to know. The most important is the current firmware version for a device. There is no specification for this as far as I can tell. ColorHug has a HID command GET_VERSION which returns 3 uint16M numbers for the major, minor and micro versions and other device firmwares have other similarly obvious but different ways of doing it.

The other is how to switch the device in firmware mode back into bootloader mode so that it can flash a new version. For ColorHug there’s a RESET command, but on other hardware it’s either a custom command sequence, or doing something physical like pressing a secret button with a paperclip or shorting two pins on a PCB.

I think it would be useful to notify the user that there in an update available, even if we can’t actually do the upgrade without doing some manual step. For this we need to get the current firmware version, ideally without open()ing the device as this will prevent other software from using it straight away. What we can get from the device for free is the device descriptors.

What I’m going to do for ColorHug is to change the unused device serial string descriptor to “FW:1.2.3”. I’ll also support in fwupd devices changing the product string from “Widget” to “Widget FW:1.2.3” i.e. we look in the various strings for a token with a “FW:” prefix and use that.

If that isn’t specified then we can fall back to opening the device and doing a custom command, but when you can ask friendly upstream firmware vendors to make a super small change, it makes things much easier for everyone. Comments welcome.

If you’re a hardware vendor reading this, I’d really like to help you get your firmware updates working on Linux. Using fwupd we can already update device firmware using UEFI capsules, and also update the various ColorHug devices. I’d love to increase the types of devices supported, so if you’re interested please let me know. Thanks!

If you’re having a baby in London, you might want to ask my wife for some help. She’s started offering one to one HypnoBirthing classes for mothers to be and birth partners designed to bring about an easier, more comfortable birthing experience.

It worked for us, and I’m super proud she’s done all the training so other people can have such a wonderful birthing experience.

Recently I spent about a day installing Fedora 22 + jhbuild on a Chromebook and left it unplugged overnight. The next day I turned it on with a flat battery, grabbed the charger, and the coreboot bios would not let me do the usual ctrl+L boot-to-SeaBIOS trick. I had to download the ChromeOS image to an SD card, reflash the ChromeOS image and thet left me without any of my Fedora workstation I’d so lovingly created the day before. This turned a $1500 laptop with a gorgeous screen into a liability that I couldn’t take anywhere for fear of losing all my work, again. The need to do CTRL+L every time I rebooted was just crazy.

I didn’t give up that easily; I need to test various bits of GNOME on a proper HiDPI screen and having a loan machine sitting in a bag wasn’t going to help anyone. So I reflashed the BIOS, and now have a machine that boots straight into Fedora 22 without any of the other Chrome stuff getting in the way.

Reflashing a BIOS on a Chromebook Pixel isn’t for the feignt of heart, but this is the list of materials you’ll need:

• Set of watchmakers screwdrivers
• Thin plastic shim (optional)
• At least 1Gb USB flash drive
• An original Chromebook Pixel
• A BIOS from here for the Pixel
• A great big dollop of courage

This does involve deleting the entire contents of your Pixel, so back anything up you care about before you start, unless it’s hosted online. I’m also not going to help you if you brick your machine, cateat emptor and all that. So, lets get cracking:

• Boot chromebook into Recovery Mode (escape+refresh at startup) then do Control+D, then Enter, wait for ~5 mins while the Pixel reflashes itself
• Power down the machine, remove AC power
• Remove the rubber pads from the underside of the Pixel, remove all 4 screws
• Gently remove the adhesive from around the edges, and use the smallest shim or screwdriver you have to release the 4 metal catches from the front and sides. You can leave the glue on the rear as this will form a hinge you can use. Hint: The tabs have to be released inwards, although do be aware there are 4 nice lithium batteries that might kinda explode if you slip and stab them hard with a screwdriver.
• Remove the BIOS write protect screw AND the copper washer that sits between the USB drives and the power connector. Put it somewhere safe.
• Gently close the bottom panel, but not enough for the clips to pop in. Turn over the machine and boot it.
• Do enough of the registration so you can logon. Then logout.
• Do the CTRL+ALT+[->] (really F2) trick to get to a proper shell and login as the chromos user (no password required). If you try to do it while logged in via the GUI it will not work.
• On a different computer, format the USB drive as EXT4 and copy the squashfs.img, vmlinuz and initrd.img files there from your nearest Fedora mirror.
• Also copy the correct firmware file from johnlewis.ie
• Unmount the USB drive and remove
• Insert the USB drive in the Pixel and mount it to /mnt
• Make a backup of the firmware using /usr/sbin/flashrom -r /mnt/backup.rom
• Flash the new firmware using /usr/sbin/flashrom -w /mnt/the_name_of_firmware.rom
• IMPORTANT: If there are any warnings or errors you should reflash with the backup; if you reboot now you’ll have a $1500 brick. If you want to go back to the backup copy just use /usr/sbin/flashrom -w /mnt/backup.rom, but lets just assume it went well for now.
• /sbin/shutdown -h now, then remove power again
• Re-open the bottom panel, which should be a lot easier this time, and re-insert the BIOS write washer and screw, but don’t over-tighten.
• Close the bottom panel and insert the clips carefully
• Insert the 4 screws and tighten carefully, then convince the sticky feet to get back into the holes. You can use a small screwdriver to convince them a little more.
  Power the machine back on and it will automatically boot to the BIOS. Woo! But not done yet.
• It will by default boot into JELTKA which is “just enough Linux to kexec another”.
• When it looks like it’s hung, enter “root” then enter and it’ll log into a root prompt.
• Mount the USB drive into /mnt again
• Do something like kexec -l /mnt/vmlinuz --initrd=/mnt/initrd.img --append=stage2=hd:/dev/sdb1:/squashfs.img
• Wait for the Fedora installer to start, then configure a network mirror where you can download packages. You’ll have to set up Wifi before you can download package lists.

This was all done from memory, so feel free to comment if you try it and I’ll fix things up as needed.