Reducing the effectiveness of a safety feature

We just purchased a 2021 KIA eNIRO to use as our family car. As typical with EVs, this has to produce a fake engine noise to avoid squashing the hard of hearing (or animals) not expecting two tons of metal to be silently moving. When we test drove a 2020 car last year, there was a physical button on the dash to turn this off, on the premise that the noise sometimes isn’t required or appropriate, but it always defaulted to “on” for every start. As the car gets faster the noise also increases in volume, until after about 30km/h fading to nothing. In reverse the same thing happens with some additional beeps. Getting our 2021 car this year, the button was no longer present and the less-than-affectionately known VESS module cannot be muted. I can guess why, someone probably turned it off and squashed something or someone and someone in the UK/US/EU government understandably freaked out. KIA also removed the wire from the wiring loom, and won’t sell the 2020 button cluster, so it’s not even like you can retrofit a new car to act like the old one.

To be super clear: I don’t have a problem with the VESS noise, but because the “speaker” is in the front bumper the solution for going backwards is “turn up the volume”. Living in London means that houses are pretty close together and me reversing into the drive at 2mph shouldn’t submit the house opposite with a noise several times louder than a huge garbage truck. The solution in the various KIA owner forums seems to be “just unplug the VESS module” but this seems at best unethical and probably borderline illegal given it’s a device with the express purpose of trying to avoid hurting someone with your 2 ton lump of metal.

Given VESS is, as you might expect, just another device on the CAN bus and people have reverse engineered the command stream so you can actually just plug the VESS module into a USB device (with a CAN converter) and play with it yourself. My idea would be to make a modchip-like device that plugs into the VESS module using the existing plug, and basically MITM the CAN messages. All messages going from VESS back to the ECU get allow-listed (even though the ECU doesn’t seem to care if the VESS goes AWOL…) and any speed measurements going forward also get passed straight through. The clever part would be to MITM the speed when the “reverse gear” command has been issued, so that the car thinks it’s going about 20km/h backwards. This makes the VESS still make the engine and beeping noise but it’s only about as loud as the VESS module when going forwards when outside the car.

Technically this is quite easy, VESS->txrxCAN->MCU->txrxCAN->ECU and you can probably use an inexpensive Microchip reference board for a prototype. My question is more if this would:

  1. Be ethical
  2. Be legal
  3. Invalidate my insurance
  4. Invalidate the warranty of my nice new shiny car

Feedback very welcome!

fwupd 1.6.0

I’ve just released the first release of the 1.6.x series, and since 1.5.x some internal plugin API has been changed and removed. Although we’ve tested this release on all the hardware we have regression tests for, bugs may have crept in; please report failures to the issue tracker as required.

There are several new plugins adding support for new hardware and a lot of code has been migrated to the new plugin API. The public libfwupd API also has some trivial additions, although no action is required.

This release adds the following notable features:

  • Add a composite ID that is used to identify dock device components
  • Add an Intel Flash Descriptor parser
  • Add API to allow the device to report its own battery level
  • Add API to refount why the the device is non-updatable
  • Add lspcon-i2c-spi programmer support
  • Add more hardware support to the pixart-rf plugin
  • Add some more new category types for firmware to use
  • Add support for downloading the SPI image from the Intel eSPI device
  • Add support for some Analogix hardware
  • Add support for writing SREC firmware
  • Add the firmware-sign command to fwupdtool to allow resigning archives
  • Split UEFI EFI binary into a subproject
  • Use an OFD or Unix lock to prevent more than one fwupdtool process

This release fixes the following bugs:

  • Actually write the bcm57xx stage1 version into the file
  • Add option to disable the UEFI capsule splash screen generation
  • Avoid use-after-free when specifying the VID/PID in dfu-tool
  • Cancel the GDBusObjectManager operation to fix a potential crash
  • Check PixArt firmware compatibility with hardware before flashing
  • Do not check for native dependencies as target dependencies
  • Do not use help2man to build manual pages
  • Fix a crash when shutting down the daemon
  • Fix build on musl
  • Fix build when using BSD
  • Fix /etc/os-release ID_LIKE field parsing
  • Force the synaptics-rmi hardware into IEP mode as required
  • Never allow D-Bus replacement when a firmware update is in operation
  • Offer the user to refresh the remote after enabling
  • Remove unused, unsafe and deprecated functions from libfwupdplugin
  • Simplify asking the user about reviews
  • Write BMP data directly without using PIL
  • Write synaptics-rmi files with valid checksum data

From a Fedora point of view, I’m waiting for the fwupd-efi new package to be reviewed and approved, and then I’ll upload the new version for Rawhide only. I might put 1.6.x into Fedora 34 after a couple more minor releases, but at the moment I’m keeping it with the 1.5.x stable branch which has all the important fixes backported. There’s a lot of new code in 1.6.x which needs to settle.

fwupd 1.5.6

Today I released fwupd 1.5.6 which the usual smattering of new features and bugfixes. These are some of the more interesting ones:

With the help of a lot of people we added support for quite a bit of new hardware. The slightly odd GD32VF103 as found in the Longan Nano is now supported, and more of the DFU ST devices with huge amounts of flash. The former should enable us to support the Pinecil device soon and the latter will be a nice vendor announcement in the future. We’ve also added support for RMI PS2 devices as found in some newer Lenovo ThinkPads, the Starlabs LabTop L4 and the new System76 Keyboard. We’ve refactored the udev and usb backends into self contained modules, allowing someone else to contribute new bluetooth peripheral functionality in the future. There are more than a dozen teams of people all working on fwupd features at the moment. Exciting times!

One problem that has been reported was that downloads from the datacenter in the US were really slow from China, specifically because the firewall was deliberately dropping packets. I assume compressed firmware looks quite a lot like a large encrypted message from a firewalls’ point of view, and thus it was only letting through ~20% of the traffic. All non-export controlled public firmware is now also mirrored onto the IPFS, and we experimentally fall back to peer-to-peer downloads where the HTTP download failed. You can prefer IPFS downloads using fwupdmgr --ipfs update although you need to have a running ipfs daemon on your local computer. If this works well for you, let me know and we might add support for downloading metadata in the future too.

We’ve fully integrated the fwupd CI with oss-fuzz, a popular fuzzing service from Google. Generating horribly corrupt firmware files has found a few little memory leaks, files that cause fwupd to spin in a loop and even the odd crash. It was a lot of work to build each fuzzer into a small static binary using a 16.04-based container but it was well worth all the hard work. All new PRs will run the same fuzzers checking for regressions which also means new plugins now also have to implement building new firmware (so the test payload can be a few tens of bytes, not 32kB), rather than just parsing it.

On some Lenovo hardware there’s a “useful” feature called Boot Order Lock that means whatever the OS adds as a BootXXXX entry the old bootlist gets restored on next boot. This breaks firmware updates using fwupdx64.efi and until we can detect BOL from a kernel interface we also check if our EFI entry has been deleted by the firmware on next boot and give the user a more helpful message than just “it failed”. Also, on some Lenovo hardware we’re limiting the number of UEFI updates to be deployed on one reboot as they appear to have slightly quirky capsule coalesce behavior. In the same vein we’re also checking the system clock is set approximately correct (as in, not set to before 2020…) so we can tell the user to check the clock on the machine rather than just failing with a obscure certificate error.

Now there are systems that can be switched to coreboot (and back to EDK2 again) we’ve polished up the “switch-branch” feature. We’re also checking both BIOSWE and BLE before identifying systems that can be supported. We’re also including the lockdown status in uploaded UEFI reports and added SBAT metadata to the fwupd EFI binary, which will be required for future versions of shim and grub – so for distro fwupd binaries the packager will need to set meson build options like -Defi_sbat_distro_id=. There are examples in the fwupd source tree.

fwupd 1.5.5

I’ve just released fwupd 1.5.5 with the following new features:

  • Add a plugin to update PixArt RF devices; the hardware this enables we’ll announce in a few weeks hopefully
  • Add new hardware to use the elantp (for TouchPads) and rts54hid (for USB Hubs) plugins
  • Allow specifying more than one VendorID for a device, which allows ATA devices to use the OUI-assigned vendor if set
  • Detect the AMD TSME encryption state for HSI-4 — use fwupdmgr security --force to help test
  • Detect the AMI PK test key is not installed for HSI-1 — a failure here is very serious
  • As usual, this release fixes quite a few bugs too:

  • Fix flashing a fingerprint reader that is in use; in theory the window to hit this is vanishingly small, but on some hardware we ask the user to authorise the request using the very device that we’re trying to update…
  • Fix several critical warnings when parsing invalid firmware, found using hongfuzz, warming my office on these cold winter days
  • Fix updating DFU devices that use DNLOAD_BUSY which fixes fwupd on some other future hardware support
  • Ignore the legacy UEFI OVMF dummy GUID so that we can test the dbx updates using qemu on older releases like RHEL
  • Make libfwupd more thread safe to fix a crash in gnome-software — many thanks to Philip Withnall for explaining a lot of the GMainContext threading complexities to me
  • We now never show unprintable chars from invalid firmware in the logs — as a result of fuzzing insane things the logs would often be full of gobbledygook, but no longer
  • I’m now building 1.5.5 into Fedora 33 and Fedora 32, packages should appear soon.

    fwupd 1.5.3

    Today we released fwupd 1.5.3 which has the usual smattering of fixes and enhancements you’d expect. One notable fix is that we now allow setting the GMainContext when used for sync methods, as some people reported problems with the way we implemented the sync libfwupd methods in previous 1.5.x releases. We’re now defaulting to the default thread context allowing the app to override if required, which seems to fix a lot of problems.

    We’ve also merged some support code to support PS/2 devices. This included adding the device firmware ID for serio class hardware. In general I’m happy to help vendors with patches that affect the core parts of fwupd (e.g. things inside ./libfwupdplugin or ./src) but plugins themselves should now either be written by the IHV or by a consulting company employed by the IHV, ODM or OEM. There are now dozens of companies adding support for new hardware all at the same time and although I’m happy to review code, I still can’t write it all :) There are lots of consulting companies to choose from now.

    We’ve also added back some compatibility code that allows apps running with old shared versions of libfwupd talk to a new running fwupd daemon – which shouldn’t be possible but then Snap decided to do just that and everything exploded.

    Aleksander also merged a few patches to fix firmware updates over QMI on some hardware and to validate firmware update method combinations on mobile broadband hardware.

    In the last release we also switched from libsoup to curl, but accidentally broke the RHEL build as it doesn’t quite have a new enough libcurl for us to use. There is now fallback code in place for these older versions.

    fwupd 1.5.2

    The last few posts I did about fwupd releases were very popular, so I’ll do the same thing again: I’ve just tagged fwupd 1.5.2 – This release changes a few things:

  • Add a build time flag to indicate if packages are supported – this would be set for “traditional” package builds done by the distro, and unset by things like the Fedora COPR build, the Flatpak or Snap bundles. There are too many people expecting that the daily snap or flatpak packages represent the “official fwupd” and we wanted to make it clear to people using these snapshots that we’ve done basically no QA on the snapshots.
  • A plugin for the Pinebook Pro laptop has been added, although it needs further work from PINE64 before it will work correctly. At the moment there’s no way of getting the touchpad version, or finding out which keyboard layout is installed so we can tag the correct firmware file. It’s nearly there and is still very useful for playing with the hardware on the PB Pro.
  • Components can now set the icon from the metadata from the LVFS, if supported by the fwupd plugin. This allows us to tag “generic” ESRT devices as things like EC devices, or, ahem, batteries.
  • I’ve been asked by a few teams, including the Red Hat Edge team, the CoreOS team and also by Google to switch from libsoup to libcurl for downloading data – as this reduces the image size by over 5MB. Even NetworkManager depends on libcurl now, and this seemed like a sensible thing to do given fwupd is now being used in so many different places.
  • Fall back to FAT32 internal partitions for detecting ESP, as some users were complaining that fwupd did not properly detect their ESP that didn’t have the correct partition GUID set. Although I think fixing the GUID is the right thing to do, the system firmware also falls back, and pragmatically so should we.
  • Fix detection of ColorHug version on older firmware versions, which was slightly embarrassing as ColorHug is one of the devices in the device regression tests, but we were not testing an old enough firmware version to detect this bug.
  • Fix reading BCM57XX vendor and device ids from firmware – firmware for the Talos II machine is already on the LVFS and can replace the non-free firmware there in almost all situations now.
  • For this release we had to improve synaptics-mst reliability when writing data, which was found occasionally when installing firmware onto a common dock model. A 200ms delay is the difference between success and failure, which although not strictly required seemed pragmatic to add.
  • Fix replugging the MSP430 device which was the last device that was failing a specific ODM QA. This allows us to release a ton of dock firmware on the LVFS.
  • Fix a deadlock seen when calling libfwupd from QT programs. This was because we were calling a sync method from threads without a context, which we’ve now added.
  • In 1.5.0 we switched to the async libfwupd by default, and accidentally dropped the logic to only download the remote metadata as required. Most users only need to download the tiny .jcat file every day, and the much larger .xml.gz is only downloaded if the signature has changed in the last 24h. Of course, it’s all hitting the CDN, but it’s not nice to waste bandwidth for no reason.
  • As Snap is bundling libfwupd with gnome-software now, we had to restore recognizing GPG and PKCS7 signature types. This allows a new libfwupd to talk to an old fwupd daemon which is something we’d not expected before.
  • We’re also now setting the SMBIOS chassis type to portable if a DeviceTree battery exists, although I’d much rather see a ChassisType in the DT specification one day. This allows us to support HSI on platforms like the PineBook Pro, although the number of tests is still minimal without more buy-in from ARM.
  • We removed the HSI update and attestation suffixes; we decided they complicated the HSI specification and didn’t really fit in. Most users won’t even care and the spec is explicitly WIP so expect further changes like this in the future.
  • If you’re running 1.5.0 or 1.5.1 you probably want to update to this release now as it fixes a hard-to-debug hang we introduced in 1.5.0. If you’re running 1.4.x you might want to let the libcurl changes settle, although we’ve been using it without issue for more than a week on a ton of hardware here. Expect 1.5.3 in a few weeks time, assuming we’re all still alive by then. :)

    New fwupd 1.5.1 release

    Hot on the heels of 1.5.0, I’ve just tagged and uploaded fwupd 1.5.1. Most importantly, if fixes the regression we recently included for an as-yet-unnamed OEM who wants to ship dock firmware. Any day now, I promise.

    Other interesting things we fixed:

    • Delete unused EFI variables when deploying firmware — which frees up a lot of space if you’ve ever enabled the fwupdx64.efi debugging…
    • Fix duplicate probe warning for the Logitech Unifying device — which was really cosmetic, but wasting resources is never nice.
    • Include the amount of NVRAM size in use in the LVFS failure report — which will might let us explain some of the dbx updates failing.
    • Make bcm57xx hotplug more reliable — although uncommon to hotplug PCI devices, using an eGPU enclosure (like I do for the device tests!) this needs to work!
    • Recognize authorized ThunderBolt value of 2 — which we found in the wild recently.
    • Remove the duplicate parent-child data in FwupdDevice and FuDevice — although not strictly a bugfix, duplicating this data made no sense and caused confusion.
    • Use UDisks to find out if swap files and devices are encrypted — which further adds more code depending on UDisks. I’ve added a Recommends: udisks2 in the Fedora package, but see the wiki if you’re running a minimal system.

    As before, Fedora 33 and 32 updates in the usual places.

    New fwupd 1.5.0 release

    Today we tagged the 1.5.0 release of fwupd. Quite a bit has changed since the last release and I figured a blog post probably made sense to explain things.

    From a firmware engineer point of view, the most useful is the ability to build composite images, for instance building a firmware.dfuse file from different A.dfu and B.dfu images. At the moment there are commands in fwupdtool to convert one file format to another, but not to merge or alter them. Many firmware files are really just containers which can store multiple images, each with optional id, index and addresses. This new fwupd feature also allows us to create very small complicated container binaries for fuzzing.

    This can be used by writing a `firmware.builder.xml` file like:

     <?xml version="1.0" encoding="UTF-8"?>
     <firmware gtype="FuBcm57xxFirmware">
       <version>1.2.3</version>
       <image>
         <version>4.5.6</version>
         <id>header</id>
         <idx>456</idx>
         <addr>0x456</addr>
         <filename>header.bin</filename>
       </image>
       <image>
         <version>7.8.9</version>
         <id>payload</id>
         <data>aGVsbG8=</data>
       </image>
     </firmware>
    

    …and then using something like fwupdtool firmware-convert firmware.builder.xml firmware.dfu builder dfu on the CLI.

    Notably, each subclass of FuFirmware (for instance FuBcm57xxFirmware) can define properties it expects in the XML so it can really be quite expressive and useful.

    From the developer point of view, the most interesting additions are the async API to libfwupd and also the addition of FwupdPlugin so we can convey enumerated system errors to the end user. This means we can finally stop the workaround of building “dummy devices” with the update error set for a generic plugin failure, e.g. efivarfs not being mounted. Expect updates to GNOME Software and GNOME Firmware to support both when all this hits Fedora stable.

    From the end user point of view, we have lots of new devices supported, including:

    • Goodix fingerprint sensors
    • Elan Touchpads
    • ChromeOS Quiche and Gingerbread
    • Broadcom BCM5719 network adapter

    The latter being the most interesting, as the BCM5719 has two branches of firmware; one from Broadcom and one free software re-implementation from meklort. Once both versions of firmware has been uploaded to the LVFS, the user can simply type fwupdmgr switch-branch to switch from the proprietary firmware to the free software one, or back again. We’re hoping to use this in other places in the future, for instance EDK2 to Coreboot on platforms without BootGuard enabled.

    Which brings me nicely to the Host Security ID. We’re not officially launching the HSI specification yet, as we’re waiting to hear back from various silicon vendors about how (and if) they can support the new initiative. HSI is something that might be hugely interesting to users where platform security is important and especially for people specifying and purchasing hardware for a specific purpose. Although fwupdmgr security is now available, you‘ll need to use --force as it’s not officially an API stable “thing” yet. If you do play with HSI, be sure to upload results to the LVFS if you can and then we’ll know if the various plugins are working as designed. I’m sure we’ll be talking more about HSI in the future…

    We’ve also done some work for teams building fwupd into products we never imagined; for CoreOS the ModemManager and flashrom plugins are split off as sub-packages so that we don’t drag lots of extra deps onto the minimal image. We’ve also made PolicyKit optional at build time as it doesn’t make sense on super-embedded devices, although you’re limited to only installing signed firmware. For the server SSH-only case we’re also using pkttyagent to request user passwords if running without GUI.

    Finally, it was a ton of work testing and fixing timing bugs for composite devices found in various laptop docks, so the people waiting for those updates probably want to update to 1.5.0 too. Updates are already on the LVFS and will be available soon. You know who you are.

    As usual, tarball releases are in the normal place and are available as a Fedora 33 update and Fedora 32 update too. Please let us know if you have any problems with 1.5.0 in the issue tracker.

    20 Million Downloads from the LVFS

    A few hours ago the LVFS provided its 20 millionth firmware update and although it’s just another somewhat unusual base-10 number, it’s an achievement I’m immensely proud of. As one of my friends said last week, “20 million of anything is a big deal”. Right from the start, the fwupd daemon and LVFS website data provider was a result of collaboration between many different companies and open source projects, and is now cemented as an integral part of the firmware ecosystem. People building open source projects, especially low level infrastructure like this, are not good at celebrating success and it’s no wonder so many talented maintainers burn out over long years of dedicated service. This post celebrates some of the things we’ve done.

    Little known to most people, fwupd and the LVFS grew out of the frustration of distributing the ColorHug firmware. If you bought one of those devices all those years ago, you can know you were a tiny part in starting all this. I still use ColorHug devices for all kinds of automated firmware testing, perhaps even more so than for screen calibration. My experience building OpenHardware devices really pushed me to make the LVFS free-for-all, on the logic that I wouldn’t have been able to justify even a $100/year subscription. Certainly making the service free in all respects meant that it was almost risk-free for companies to test the service.

    Now the LVFS analyses uploaded firmware for security problems, keeps millions of devices up to date, and also helps governments buy secure hardware using initiatives like the upcoming Host Security ID that I’ll talk more about in future blog posts. How many devices we’ve updated is impossible to know exactly as many large companies and departments mirror the entire LVFS; we just know it’s at least 20 million. In reality it’s probably a single digit multiple of that, although there’s no real way of knowing. We know 1.5 million people have sent the optional “it works for me” report we ask from CLI users, and given the CLI downloads account for ~1% of all downloads it could be a lot higher than 20 million.

    A huge number of devices are supported on the LVFS now. There are currently 2393 different public firmwares uploaded by 1401 users from 106 different vendors, using 39 different protocols to update hardware. We’ve run 40,378 automated tests on those files, and extracted 1,170,543 file volume objects which can be scanned by YARA. All impressive numbers I’m sure you’ll agree.

    There is one specific person I would like to thank first, my co-maintainer for both projects: Mario Limonciello who is a Senior Principal Software Development Engineer at Dell. Mario has reviewed thousands of my patches over the years, and contributed hundreds himself. I really appreciate his trust and also his confidence to tell me the half-baked and incomplete thing I’m proposing is actually insane. Together we’ve created an architecture that’s easy to maintain with a clean and modern design. Dell were also the first major laptop OEM to tell their suppliers “you need to ship firmware updates using the LVFS” and so did a lot of the initial plumbing work. We re-used most of the initial plugins when other OEMs decided they’d like to join the initiative later.

    Peter Jones is another talented member of our team at Red Hat and wrote a lot of the low level UEFI code we’ve used millions of times. Peter has to understand all the crazy broken things that firmware vendors decide to do, and is responsible for most of the EFI code in fwupd. Over the years fwupd has absorbed two of his projects, fwupdate and most recently dbxtool. Without Peter there would have been no UpdateCapsule support, and that’s about half the updates on the LVFS.

    Also notable to mention here is Logitech. They’ve shipped a ton of firmware (literally, millions) for their Unifying hardware and were also early adopters of the LVFS and fwupd. Nestor Lopez Casado, many thanks for all your help over the years and I’m glad MouseJack made all this a requirement :)

    I’d also like to thank Lenovo; not a specific person, as Lenovo is split up into ThinkPad, ThinkStation and ThinkCenter groups and in each the engineers would probably like to remain anonymous. Lenovo as a combined group is shipping a huge amount of firmware now via the LVFS, and most of the Lenovo supply chain is already wired up to supporting the LVFS. A lot of the ODMs for Lenovo have had to actually install Fedora and learn how to program with GLib C to create a fwupd plugins to support laptop models that are not even on the shelves yet. Training up dozens of people in “how to write a fwupd plugin and deal with a grumpy maintainer” took a lot of time, but now we have companies building custom silicon submitting ready-to-roll plugins, fixes and even enhancements to the GUI tools.

    The LVFS isn’t just a way for OEMs to distribute firmware, like a shared FTP site. The LVFS plumbs-itself into the ODM and ISV relationships, so we can get a pipeline right from the firmware author, all the way to the end user. As ODMs such as Wistron and Foxconn use the LVFS for one OEM, it’s very simple for them to also support other OEMs. The feedback loop from vendors to users and back to vendors again has been invaluable when debugging problems with specific firmware releases.

    More recently various groups at Google have also been pushing suppliers of the Chromebook ecosystem to use fwupd and the LVFS. I’ve been told there’s now a “fwupd group” inside Google and I know that there are more than a few different models that will only be updatable using fwupd. Google are also using some of the consulting companies familiar with LVFS and fwupd so that it’s not just me explaining how this works over and over to various different small ODMs. I think in the next year this consulting side will explode and help grow the ecosystem even further.

    I’d love to list all the OEMs, ODMs, ISVs and consultants that have helped over the last 5 years, but this blog entry would be even larger than it already is. Just know that I appreciate your support, help and guidance.

    Talk of growing the ecosystem, the Linux Foundation are taking over the actual site maintenance; I’m not a sysadmin, and Terraform and Docker still scare me. This Christmas we’ll move the little VM I have running in Amsterdam to a proper scalable architecture with 24/7 support. This will let us provide some kind of uptime guarantee and also means I don’t have to worry about applying a security update when I’m on holiday without internet access. The Linux Foundation have been paying the entire CDN cost for the last few years, and for that my wallet is truly grateful.

    Finally, I must also thank Red Hat for letting me work on this stuff. Over the last few years fwupd has gone from my “20%” hobby to almost taking over all my developer time. Red Hat doesn’t get enough credit for all the essential plumbing they tirelessly do, and without them paying my salary every month there is no way this kind of free-to-upload and free-to-download service could exist. I firmly believe that fwupd is mutually beneficial to all the Red Hat partners like Intel, Dell and Lenovo – both so that more hardware gets purchased from OEMs, and also so that customers running Fedora and RHEL have up to date firmware.

    At a conference last year, I presented a talk where the penultimate slide was “the LVFS is just a website that runs cron jobs” and I had someone I respect come up to me afterwards and tell me something in a stern voice I’ll remember forever: “You didn’t just create a website – you changed an industry!

    Lets look forward to the next 20 million updates.

    Updating Secure Boot dbx with fwupd and the LVFS

    This is one of those blog posts which is going to escalate quickly; in my effort to make it understandable I might simplify some of the theory, so if you know all this stuff already please scroll down a few paragraphs and try not to be pedantic.

    UEFI Secure Boot is protection technology that is designed to protect a system against malicious code being executed early in the boot process. It defines a way to “lock down” the platform so only binaries signed with a specific key will run. The certificate authority is managed by Microsoft, which is unimportant generally, but was hugely important before Microsoft agreed to sign loaders like shim as it meant hardware wouldn’t run Linux. SecureBoot isn’t evil by itself, and having SecureBoot turned on protects your hardware from real world attacks and in theory secures most of the boot process to where the Linux kernel takes over. In the case where you don’t love or trust Microsoft you can configure another certificate authority to manage the trust somewhere else, but that’s a blog post for another day.

    Anyway — I alluded that Microsoft has a special process for signing our Linux binaries so we could still keep running Linux on hardware that can’t turn off SecureBoot for one reason or another. What happened with the BootHole set of CVEs was that a researcher from Eclypsium found a nasty bug in the GRUB project (of which the binary was signed by Microsoft) which allowed complete circumvention of the SecureBoot architecture. This would mean a bootkit that previously would only work with SB turned off (which is fairly uncommon, as Microsoft forces OEMs to turn it on by default) now could be easily deployed onto hardware with SB enabled using an old version of shim packaged in the bootkit.

    Luckily, this scenario was planned for by the UEFI people, and as well as having a “binaries must be signed by this certificate” functionality we also have a “the binary must not have this checksum” back up plan. Microsoft recently invoked the back-up-plan and added quite a few checksums of binaries like shim and grub to the latest dbx update. They’ve done this three times since 2010, but only traditionally for things like the Symantec recovery binary which most people won’t have installed. The signed “blocklist of checksums” is called the dbx internally, and can only be updated and can never be downgraded for obvious reasons.

    If you actually deploy the current UEFI Revocation List (a.k.a dbx) on your Linux box right now using something like dbxtool it’ll most probably apply fine, and then when you reboot you’ll get a nice red screen and a SECURE BOOT VIOLATION message, and then for most people the computer will be a useless brick. The grub and shim installed into /boot/efi by your OS is now being blocked, and so nothing works. You can’t even boot a Linux installer to reinstall as the grub on the live media is going to be blocked too.

    What we used to do was have a dbxtool.service that just applied the latest dbx (if you were not using a LiveCD) and then just hope it all still boots. This mostly worked, as the dbx contained checksums of things you were not likely to be using, rather than things you are most likely to have. You might also be thinking this also isn’t an awesome way to deploy a single-use binary to millions of computers. It’s like shipping firmware updates in an rpm. Hmm. That gives me an idea.

    What we could do is have a fwupd plugin that reads the current dbx and creates a device:

    │   
    └─UEFI dbx:
          Device ID:           362301da643102b9f38477387e2193e57abaa590
          Summary:             UEFI Revocation Database
          Current version:     77
          Minimum Version:     77
          Install Duration:    1 second
          GUIDs:               5971a208-da00-5fce-b5f5-1234342f9cf7 ← UEFI\CRT_A9087D1044AD18F7A94916D284CBC01827CF23CD8F60B79072C9CAA1FEF4D649&ARCH_X64
                               f8ba2887-9411-5c36-9cee-88995bb39731 ← UEFI\CRT_A1117F516A32CEFCBA3F2D1ACE10A87972FD6BBE8FE0D0B996E09E65D802A503&ARCH_X64
          Device Flags:        • Internal device
                               • Updatable
                               • Supported on remote server
                               • Needs a reboot after installation
    

    Then we could push the dbx updates onto the LVFS so that they get downloaded as required rather than stored into /usr for all eternity. We could put all the checks into fwupd to verify that the user doesn’t have any blocked binaries into the ESP (as fwupd already knows how to mount the ESPs to deploy UEFI capsules) so that we don’t accidentally brick any systems:

    And we could wire this up to the GUI so that it’s super easy to use, and makes it a one-click install to deploy locally or scriptable using the CLI:

    If you install fwupd and gnome-software from git master, this is exactly what you can do right now. To test this on your non-production system, you can add something like this to /etc/fwupd/remotes.d/dbx.conf:

    [fwupd Remote]
    Enabled=true
    Title=UEFI dbx
    Keyring=gpg
    MetadataURI=https://fwupd.org/downloads/firmware-1d780afd8789afc6bccff638e5e8376604ba082aa32dc06db71d6768824a1efa.xml.gz
    ReportURI=https://fwupd.org/lvfs/firmware/report
    OrderBefore=lvfs,fwupd
    

    Then fwupdmgr refresh gets you the metadata and fwupdmgr update applies the update only if it’s safe to do so. Again: Do not do this on a system you don’t have backups for.

    I’m going to test this myself on a few more machines and then once the shim and grub builds hit Fedora stable we can think about packaging up the new fwupd so that we can deploy the dbx configuration using the LVFS easily. Of course, nobody is taking away doing it manually with dbxtool, but now it’s integrated into the system and easy for the user to deploy. Comments and corrections most welcome.