Technical Blog of Richard Hughes

Today I tagged fwupd 1.8.4 which adds a few nice features and bug fixes. One specific enhancement I wanted to shout about is that we’re now supplying translated summary, description text and suggested actions for each HSI security failure. Two of the most common criticisms of the new GNOME security panel were “but what does it mean” and also “and what should I do” which ironically were fixed long before all the hubbub erupted. If you want to see both new bits of data then make sure you’re using gnome-control-center from the main branch and then install the new fwupd version – although if you’re stuck on a distro version of fwupd GNOME will still fallback to the single-line summary line as before.

One additional new feature that might accidentally fix another criticism with the panel is that fwupd now reads your system BIOS settings, and has the ability to change them if the user desires (and has authorization to do). This means we have to match the HSI failure (e.g. IOMMU disabled) with the BIOS setting, which isn’t standardized at all between vendors. We currently support this on modern Lenovo and Dell platforms via the firmware-attributes kernel interface; other vendors just have to add the kernel WMI bridge and it should mostly magically start to work.

As we now know what the failure is, what we need to change, and how to change it, we can actually ask the user if they want to change the setting automatically in the fwupdmgr security command line. This would allow us to add a “JFDI” action in the new GNOME device security panel rather than asking the user to manually change a firmware setting in the BIOS. We won’t do this for GNOME 43 as we need a few months of real-world testing to see what attributes are 100% safe to change on actual user systems, but for GNOME 44 the panel could be a whole lot more helpful than it is now.

A new tantalizing features then become available when using fwupd, as we can now read and change firmware settings. One is the ability to emulate the BIOS settings of another machine, which is fairly uninteresting to end users, but allows us the developers to reproduce bugs much easier now that we’re doing cleverer things. One more interesting deployment feature is that we also support reading out a file from /etc and applying those firmware settings at startup. This means you can now deploy a machine using something like Ansible, and have the firmware settings set up in the same way you set up the local machine state. There are lots of docs on how this all works and I encourage you to try this out and let us know how it goes. One caveat is that this doesn’t work if you have a password set on your BIOS settings, but we’re working on this for the next version.

Needless to say, please tell us about any problems with the new release. As always, comments welcome.

As some as you may know, there might be firmware security support in the next versions of Plymouth, GNOME Control Center and GDM. This is a great thing, as most people are running terribly insecure hardware and have no idea. The great majority of these systems can be improved with a few settings changes, and the first step in my plan is showing people what’s wrong, giving some quick information, and perhaps how to change it. The next step will be a “fix the problem” button but that’s still being worked on, and will need some pretty involved testing for each OEM. For the bigger picture there’s the HSI documentation which is a heavy and technical read but the introduction might be interesting. For other 99.99% of the population here are some pretty screenshots:

To facilitate development of various UIs, fwupd now supports emulating different systems. This would allow someone to show dozens of supported devices in GNOME Firmware or to showcase the firmware security panel in the GNOME release video. Hint hint. :)

To do this, ensure you have fwupd 1.8.3 installed (or enable the COPR), and then you can do:

sudo FWUPD_HOST_EMULATE=thinkpad-p1-iommu.json.gz /usr/libexec/fwupd/fwupd

Emulation data files can be created with ./contrib/generate-emulation.py file.json in the fwupd source tree and then can be manually modified if required. Hint: it’s mostly the same output as fwupdmgr get-devices --json and fwupdmgr security --json and you can run generate-emulation.py on any existing JSON output to minimize it.

To load a custom profile, you can do something like:

sudo FWUPD_HOST_EMULATE=/tmp/my-system.json /usr/libexec/fwupd/fwupd

As a precaution, the org.fwupd.hsi.HostEmulation attribute is added so we do not ask the user to upload the HSI report. The emulated devices are also not updatable for obvious reasons. Comments welcome!

I’ve just tagged the 1.8.0 release of fwupd, with these release notes — there’s lots of good stuff there as always. More remarkable is that LVFS has now supplied over 50 million updates to Linux machines all around the globe. The true number is going to be unknown, as we allow vendors to distribute updates without any kind of logging, and also allow companies or agencies to mirror the entire LVFS so the archive can be used offline. The true number of updates deployed will be a lot higher than 50 million, which honestly blows my tiny mind. Just 7 years ago Christian asked me to “make firmware updates work on Linux” and now we have a thriving client project that respects both your freedom and your privacy, and a thriving ecosystem of hardware vendors who consider Linux users first class citizens. Of course, there are vendors who are not shipping updates for popular hardware, but they’re now in the minority — and every month we have two or three new vendor account requests. The logistical, security and most importantly commercial implications of not being “on the LVFS” are now too critical even for tier-1 IHVs, ODMs and OEMs to ignore.

I’m still amazed to see Reddit posts, YouTube videos and random people on Twitter talk about the thing that’s been my baby for the last few years. It’s both frightening as hell (because of the responsibility) and incredibly humbling at the same time. Red Hat can certainly take a lot of credit for the undeniable success of LVFS and fwupd, as they have been the people paying my salary and pushing me forward over the last decade and more. Obviously I’m glad everything is being used by the distros like Ubuntu and Arch, although for me it’s Fedora that’s at least technically the one pushing Linux forward these days. I’ve seen Fedora grow in market share year on year, and I’m proud to be one of the people pushing the exciting Future Features into Fedora.

So what happens next? I guess we have the next 50 million updates to look forward to. The LVFS has been growing ever so slightly exponentially since it was first conceived so that won’t take very long now. We’ve blasted through 1MM updates a month, and now regularly ship more than 2MM updates a month and with the number of devices supported growing like it has (4004 different streams, with 2232 more planned), it does seem an exciting place to be. I’m glad that the number of committers for fwupd is growing at the same pace as the popularity, and I’m not planning to burn out any time soon. Google has also been an amazing partner in encouraging vendors to ship updates on the LVFS and shipping fwupd in ChromeOS — and their trust and support has been invaluable. I’m also glad the “side-projects” like “GNOME Firmware“, “Host Security ID“, “fwupd friendly firmware” and “uSWID as an SBoM” also seem to be flourishing into independent projects in their own right. It does seem now is the right time to push the ecosystem towards transparency, open source and respecting the users privacy. Redistributing closed source firmware may be an unusual route to get there, but it’s certainly working. There are a few super-sekret things I’m just not allowed to share yet, but it’s fair to say that I’m incredibly excited about the long term future.

From the bottom of my heart, thank you all for your encouragement and support.

A Software Bill of Materials (aka SBoM) is something you’ve probably never heard of, but in future years they’ll hopefully start to become more and more important. In May last year the US president issued an executive order titled Improving the Nation’s Cybersecurity in which it outlines the way that critical software used by various branches of the government should be more traceable and secure. One of the key information captured in a SBoM is “who built what from where” which in open source we’re already familiar with, e.g. “Red Hat built your Linux kernel in a datacenter in the US” rather than “random person from the internet build your container on their laptop using Debian Sarge” and in the former case we also always have the hash of the source archive that was used to build it, and a lot more. Where this concept breaks down is firmware, where lots of different entities build each subsection in different ways, usually due to commercial and technical constraints.

Firmware is often lumped together as one thing, both technically as-in “one download” and conceptually when thinking about OS security. In reality a single firmware image might contain a FSP from Intel, several updated CPU microcode blobs for a few different CPUs, a CSME management engine, an embedded controller update, a UEFI system firmware a lot more. The system firmware is then made up of different file volumes, each with a few dozen EFI “PEI” binaries for initial system start-up and then a couple of hundred (!) “DXE” binaries for things like pre-boot networking and things like fingerprint authentication, mouse and keyboard input.

In the executive order from last May, firmware was explicitly excluded from the list of software that required a SBoM, on the logic that none of the infrastructure or specifications were in place, and it just wasn’t possible to do. The requirement for SBoM for boot-level firmware is expected in subsequent phases of the executive order. Needless to say I’ve been spending the last few months putting all the pieces together to make a firmware SBoM not just possible, but super easy for OEMs, ODMs and IBVs to generate.

The first problem to solve is how to embed the software ID (also known as SWID) metadata into each EFI binary. This is solved by putting coSWID metadata (a DTMF specification) into a new COFF section called, unsurprisingly, “SBOM”. This allows us to automatically capture at build time some data, for instance the tree hash, and the files that were used to build the binary, etc. This is what my friends at Eclypsium have been working on – so soon you can drop a top-level vendor.ini file in your EDK2 checkout with the correct vendor data (legal name, home page etc.) and then you can just build the tree and get everything inserted in this new PE section automatically. This gets us half way there. The uSWID readme explains how to do this manually too, for people not using either the EDK2 build-system or a variant of it.

The second problem is how to include SWID metadata for the blobs we either don’t build, or we can’t modify in any way, e.g. the FSP or uCode. For this there’s an “external” version of the same coSWID metadata which has a simple header we can find in the firmware image. This can either be included in the file volume itself, or just included as a file alongside the binary deliverable. We just have to trust that the vendor includes the correct metadata there – and we’re already trusting the vendor to implement things like SecureBoot correctly. The vendor can either use the [pip install] uswid command line (more examples in the uSWID readme) or more helpfully there’s also a web-generator on the LVFS that can spit out the tiny coSWID blob with the correct header ready to be included somewhere in the binary image.

Open source firmware like coreboot is also in the same boat of course, but here we have more flexibility in how to generate and include the SWID metadata in the image. My friends at Immune and 9elements are planning to work on this really soon, so we can have feature parity for free firmware like coreboot – even when non-free blobs are included into the image so that it can actually work on real hardware.

So, we have the metadata provision from the IBV, ODM and OEM all sprinkled around the update binary. What do we do then? When the binary is uploaded to the LVFS we decompress all the shards of the firmware, and do various checks. At this point we can look for coSWID metadata in the EFI binaries and also uSWID+coSWID metadata for the non-free blobs. From this we can save any of the detected SWID metadata to the per-component datastore, and make it available as a publicly available SBoM HTML page and also .zip archive containing the raw SWID XML data. It probably makes sense to have an external tool, either a CLI utility in the lvfs-website project, or something in native golang — but that doesn’t exist yet.

The vendor also gets the all important “green tick” which means the customer buying the hardware knows that it’s complying with the new requirements. Of course, we can’t check if the ODM has included all the SWID metadata for all the binaries, or included all the SWID components for all of the nonfree chunks, but it’s good enough as a first pass. The next logical thing would be to make a rule saying that the SWID green tick disappears if we detected CPU microcode, but also didn’t detect any microcode SWID metadata, etc. It would also be interesting to show a pie-chart for a given firmware image, showing just where the firmware has been built from, and who by, and how much stuff remains unaccounted for. But, little steps first.

I think I’ve got agreement-in-principal from most of the major stakeholders, and I’ll be hopefully presenting this work alongside AMI to the UEFI forum in a few months time. This means we’re in a position to actually provide SBoM for all firmware when the next EO revision is announced, rather than the ecosystem collapsing into a ball of raw panic.

If you want to add uSWID metadata to your firmware please let me know how I can help, even if it’s not available on the LVFS yet; I think this makes just as much sense for firmware that sits on a USB hub as it does your system firmware. Comments welcome.

Does anyone have any examples of peripheral devices that can have their firmware upgraded by dropping a new firmware file onto a mounted volume? e.g. insert device, new disk appears, firmware file is copied over, then the firmware update completes?

Could anyone with a device that supports firmware upgrade using bulk storage please fill in my 2 minute questionnaire? I’m trying to create a UF2-compatible plugin to fwupd and need data to make sure it’s suitable for all vendors and devices. The current pull request is here, but I have no idea if it is suitable yet. Thanks!

I’ve just deployed some new functionality to the LVFS adding support for component <tag>s. These are used by server vendors to identify a known-working (or commercially supported) set of firmware on the machine. This is currently opt-in for each vendor to avoid the UI clutter on the components view, and so if you’re a vendor reading this post and realize you want this feature, let me know and it’s two clicks on the admin panel.

The idea is that when provisioning the machine, we can set HostBkc=vendor-2021q1 in /etc/fwupd/daemon.conf and then any invocation of fwupdmgr sync-bkc will install or downgrade firmware on all compatible devices (UEFI, RAID, network adapter, & SAS HBA etc.) to make the system match a compatible set. This allows two things:

• Factory recovery where a system in the field has been upgraded
• Ensuring a consistent set of vendor-tested firmware for a specific workload

The tags are either assigned in the archive firmware.metainfo.xml file or added post-upload on the LVFS and are then included in the public AppStream metadata. A single firmware can be marked with multiple tags, and tags can be duplicated for different firmwares. This would allow a server vendor to say “this set of firmware has been tested as a set for workload A, and this other set of firmware has been tested for workload B” which is somewhat odd for us consumer-types, but seems to be pretty normal for enterprise deployments.

As a bonus feature, updating or downgrading firmware away from the “Best Known Configuration” is allowed, but we’ll show a semi-scary warning. Using fwupdmgr sync-bkc will undo any manual changes and bring the machine back to the BKC. Needless to say fwupd will not ship with a configured BKC.

We’ll include this somewhat-niche-but-required feature with fwupd 1.7.3 which will hopefully be released before Christmas. Questions and comments welcome.

tl;dr: if you’re using libfwupd to download firmware, nothing changes and everything continues as before. If you’re using something like wget that doesn’t follow redirects by default you might need to add a command line argument to download firmware from the LVFS.

Just a quick note to explain something that some people might have noticed; if you’re using fwupd >= 1.6.1 or >= 1.5.10 when you connect to the LVFS to download a firmware file you actually get redirected to the same file on the CDN. e.g. downloading https://fwupd/download/foo.cab gets a redirect to https://cdn.fwupd/download/foo.cab which is then streamed to the user. Why this insanity?

As some of you know, egress charges from AWS are insanely high. The Linux Foundation are the kind people that kindly pay the LVFS bill every month, and a 4 years ago that was just a few hundred dollars and that was a rounding error to them. Last year we again grew at more than 100% and the projection for next year is going to surpass even that; the average size of firmware files has gone from ~30MB to ~50MB with much, much, larger server firmware in the pipeline. We certainly can’t watch the egress bill scale linearly with the LVFS popularity, else some accountant at the Linux Foundation is going to start asking questions – especially when Fastly provides the LF a geo-replicated CDN – which we’re not using.

So why don’t we put the CDN URL in the XML metadata directly, and then avoid all this redirect complexity altogether? This time the lawyers get us, as we’re required by US law to restrict distribution of some firmware to some countries on an embargo list. It’s very complicated, and it varies by vendor, but it’s not something we can avoid. So for this reason, the LVFS does a GeoIP lookup on the client IP, and if it’s all okay we then redirect the client to the CDN-cached version. It also lets us tell the vendor how many times the firmware has been downloaded without importing the CDN logs every 24 hours – which would be even harder as we only keep them for a short time for privacy reasons.

We just purchased a 2021 KIA eNIRO to use as our family car. As typical with EVs, this has to produce a fake engine noise to avoid squashing the hard of hearing (or animals) not expecting two tons of metal to be silently moving. When we test drove a 2020 car last year, there was a physical button on the dash to turn this off, on the premise that the noise sometimes isn’t required or appropriate, but it always defaulted to “on” for every start. As the car gets faster the noise also increases in volume, until after about 30km/h fading to nothing. In reverse the same thing happens with some additional beeps. Getting our 2021 car this year, the button was no longer present and the less-than-affectionately known VESS module cannot be muted. I can guess why, someone probably turned it off and squashed something or someone and someone in the UK/US/EU government understandably freaked out. KIA also removed the wire from the wiring loom, and won’t sell the 2020 button cluster, so it’s not even like you can retrofit a new car to act like the old one.

To be super clear: I don’t have a problem with the VESS noise, but because the “speaker” is in the front bumper the solution for going backwards is “turn up the volume”. Living in London means that houses are pretty close together and me reversing into the drive at 2mph shouldn’t submit the house opposite with a noise several times louder than a huge garbage truck. The solution in the various KIA owner forums seems to be “just unplug the VESS module” but this seems at best unethical and probably borderline illegal given it’s a device with the express purpose of trying to avoid hurting someone with your 2 ton lump of metal.

Given VESS is, as you might expect, just another device on the CAN bus and people have reverse engineered the command stream so you can actually just plug the VESS module into a USB device (with a CAN converter) and play with it yourself. My idea would be to make a modchip-like device that plugs into the VESS module using the existing plug, and basically MITM the CAN messages. All messages going from VESS back to the ECU get allow-listed (even though the ECU doesn’t seem to care if the VESS goes AWOL…) and any speed measurements going forward also get passed straight through. The clever part would be to MITM the speed when the “reverse gear” command has been issued, so that the car thinks it’s going about 20km/h backwards. This makes the VESS still make the engine and beeping noise but it’s only about as loud as the VESS module when going forwards when outside the car.

Technically this is quite easy, VESS->txrxCAN->MCU->txrxCAN->ECU and you can probably use an inexpensive Microchip reference board for a prototype. My question is more if this would:

1. Be ethical
2. Be legal
3. Invalidate my insurance
4. Invalidate the warranty of my nice new shiny car

Feedback very welcome!

I’ve just released the first release of the 1.6.x series, and since 1.5.x some internal plugin API has been changed and removed. Although we’ve tested this release on all the hardware we have regression tests for, bugs may have crept in; please report failures to the issue tracker as required.

There are several new plugins adding support for new hardware and a lot of code has been migrated to the new plugin API. The public libfwupd API also has some trivial additions, although no action is required.

This release adds the following notable features:

• Add a composite ID that is used to identify dock device components
• Add an Intel Flash Descriptor parser
• Add API to allow the device to report its own battery level
• Add API to refount why the the device is non-updatable
• Add lspcon-i2c-spi programmer support
• Add more hardware support to the pixart-rf plugin
• Add some more new category types for firmware to use
• Add support for downloading the SPI image from the Intel eSPI device
• Add support for some Analogix hardware
• Add support for writing SREC firmware
• Add the firmware-sign command to fwupdtool to allow resigning archives
• Split UEFI EFI binary into a subproject
• Use an OFD or Unix lock to prevent more than one fwupdtool process

This release fixes the following bugs:

• Actually write the bcm57xx stage1 version into the file
• Add option to disable the UEFI capsule splash screen generation
• Avoid use-after-free when specifying the VID/PID in dfu-tool
• Cancel the GDBusObjectManager operation to fix a potential crash
• Check PixArt firmware compatibility with hardware before flashing
• Do not check for native dependencies as target dependencies
• Do not use help2man to build manual pages
• Fix a crash when shutting down the daemon
• Fix build on musl
• Fix build when using BSD
• Fix /etc/os-release ID_LIKE field parsing
• Force the synaptics-rmi hardware into IEP mode as required
• Never allow D-Bus replacement when a firmware update is in operation
• Offer the user to refresh the remote after enabling
• Remove unused, unsafe and deprecated functions from libfwupdplugin
• Simplify asking the user about reviews
• Write BMP data directly without using PIL
• Write synaptics-rmi files with valid checksum data

From a Fedora point of view, I’m waiting for the fwupd-efi new package to be reviewed and approved, and then I’ll upload the new version for Rawhide only. I might put 1.6.x into Fedora 34 after a couple more minor releases, but at the moment I’m keeping it with the 1.5.x stable branch which has all the important fixes backported. There’s a lot of new code in 1.6.x which needs to settle.

Today I released fwupd 1.5.6 which the usual smattering of new features and bugfixes. These are some of the more interesting ones:

With the help of a lot of people we added support for quite a bit of new hardware. The slightly odd GD32VF103 as found in the Longan Nano is now supported, and more of the DFU ST devices with huge amounts of flash. The former should enable us to support the Pinecil device soon and the latter will be a nice vendor announcement in the future. We’ve also added support for RMI PS2 devices as found in some newer Lenovo ThinkPads, the Starlabs LabTop L4 and the new System76 Keyboard. We’ve refactored the udev and usb backends into self contained modules, allowing someone else to contribute new bluetooth peripheral functionality in the future. There are more than a dozen teams of people all working on fwupd features at the moment. Exciting times!

One problem that has been reported was that downloads from the datacenter in the US were really slow from China, specifically because the firewall was deliberately dropping packets. I assume compressed firmware looks quite a lot like a large encrypted message from a firewalls’ point of view, and thus it was only letting through ~20% of the traffic. All non-export controlled public firmware is now also mirrored onto the IPFS, and we experimentally fall back to peer-to-peer downloads where the HTTP download failed. You can prefer IPFS downloads using fwupdmgr --ipfs update although you need to have a running ipfs daemon on your local computer. If this works well for you, let me know and we might add support for downloading metadata in the future too.

We’ve fully integrated the fwupd CI with oss-fuzz, a popular fuzzing service from Google. Generating horribly corrupt firmware files has found a few little memory leaks, files that cause fwupd to spin in a loop and even the odd crash. It was a lot of work to build each fuzzer into a small static binary using a 16.04-based container but it was well worth all the hard work. All new PRs will run the same fuzzers checking for regressions which also means new plugins now also have to implement building new firmware (so the test payload can be a few tens of bytes, not 32kB), rather than just parsing it.

On some Lenovo hardware there’s a “useful” feature called Boot Order Lock that means whatever the OS adds as a BootXXXX entry the old bootlist gets restored on next boot. This breaks firmware updates using fwupdx64.efi and until we can detect BOL from a kernel interface we also check if our EFI entry has been deleted by the firmware on next boot and give the user a more helpful message than just “it failed”. Also, on some Lenovo hardware we’re limiting the number of UEFI updates to be deployed on one reboot as they appear to have slightly quirky capsule coalesce behavior. In the same vein we’re also checking the system clock is set approximately correct (as in, not set to before 2020…) so we can tell the user to check the clock on the machine rather than just failing with a obscure certificate error.

Now there are systems that can be switched to coreboot (and back to EDK2 again) we’ve polished up the “switch-branch” feature. We’re also checking both BIOSWE and BLE before identifying systems that can be supported. We’re also including the lockdown status in uploaded UEFI reports and added SBAT metadata to the fwupd EFI binary, which will be required for future versions of shim and grub – so for distro fwupd binaries the packager will need to set meson build options like -Defi_sbat_distro_id=. There are examples in the fwupd source tree.