New fwupd 1.5.0 release

Today we tagged the 1.5.0 release of fwupd. Quite a bit has changed since the last release and I figured a blog post probably made sense to explain things.

From a firmware engineer point of view, the most useful is the ability to build composite images, for instance building a firmware.dfuse file from different A.dfu and B.dfu images. At the moment there are commands in fwupdtool to convert one file format to another, but not to merge or alter them. Many firmware files are really just containers which can store multiple images, each with optional id, index and addresses. This new fwupd feature also allows us to create very small complicated container binaries for fuzzing.

This can be used by writing a `firmware.builder.xml` file like:

 <?xml version="1.0" encoding="UTF-8"?>
 <firmware gtype="FuBcm57xxFirmware">
   <version>1.2.3</version>
   <image>
     <version>4.5.6</version>
     <id>header</id>
     <idx>456</idx>
     <addr>0x456</addr>
     <filename>header.bin</filename>
   </image>
   <image>
     <version>7.8.9</version>
     <id>payload</id>
     <data>aGVsbG8=</data>
   </image>
 </firmware>

…and then using something like fwupdtool firmware-convert firmware.builder.xml firmware.dfu builder dfu on the CLI.

Notably, each subclass of FuFirmware (for instance FuBcm57xxFirmware) can define properties it expects in the XML so it can really be quite expressive and useful.

From the developer point of view, the most interesting additions are the async API to libfwupd and also the addition of FwupdPlugin so we can convey enumerated system errors to the end user. This means we can finally stop the workaround of building “dummy devices” with the update error set for a generic plugin failure, e.g. efivarfs not being mounted. Expect updates to GNOME Software and GNOME Firmware to support both when all this hits Fedora stable.

From the end user point of view, we have lots of new devices supported, including:

  • Goodix fingerprint sensors
  • Elan Touchpads
  • ChromeOS Quiche and Gingerbread
  • Broadcom BCM5719 network adapter

The latter being the most interesting, as the BCM5719 has two branches of firmware; one from Broadcom and one free software re-implementation from meklort. Once both versions of firmware has been uploaded to the LVFS, the user can simply type fwupdmgr switch-branch to switch from the proprietary firmware to the free software one, or back again. We’re hoping to use this in other places in the future, for instance EDK2 to Coreboot on platforms without BootGuard enabled.

Which brings me nicely to the Host Security ID. We’re not officially launching the HSI specification yet, as we’re waiting to hear back from various silicon vendors about how (and if) they can support the new initiative. HSI is something that might be hugely interesting to users where platform security is important and especially for people specifying and purchasing hardware for a specific purpose. Although fwupdmgr security is now available, you‘ll need to use --force as it’s not officially an API stable “thing” yet. If you do play with HSI, be sure to upload results to the LVFS if you can and then we’ll know if the various plugins are working as designed. I’m sure we’ll be talking more about HSI in the future…

We’ve also done some work for teams building fwupd into products we never imagined; for CoreOS the ModemManager and flashrom plugins are split off as sub-packages so that we don’t drag lots of extra deps onto the minimal image. We’ve also made PolicyKit optional at build time as it doesn’t make sense on super-embedded devices, although you’re limited to only installing signed firmware. For the server SSH-only case we’re also using pkttyagent to request user passwords if running without GUI.

Finally, it was a ton of work testing and fixing timing bugs for composite devices found in various laptop docks, so the people waiting for those updates probably want to update to 1.5.0 too. Updates are already on the LVFS and will be available soon. You know who you are.

As usual, tarball releases are in the normal place and are available as a Fedora 33 update and Fedora 32 update too. Please let us know if you have any problems with 1.5.0 in the issue tracker.

20 Million Downloads from the LVFS

A few hours ago the LVFS provided its 20 millionth firmware update and although it’s just another somewhat unusual base-10 number, it’s an achievement I’m immensely proud of. As one of my friends said last week, “20 million of anything is a big deal”. Right from the start, the fwupd daemon and LVFS website data provider was a result of collaboration between many different companies and open source projects, and is now cemented as an integral part of the firmware ecosystem. People building open source projects, especially low level infrastructure like this, are not good at celebrating success and it’s no wonder so many talented maintainers burn out over long years of dedicated service. This post celebrates some of the things we’ve done.

Little known to most people, fwupd and the LVFS grew out of the frustration of distributing the ColorHug firmware. If you bought one of those devices all those years ago, you can know you were a tiny part in starting all this. I still use ColorHug devices for all kinds of automated firmware testing, perhaps even more so than for screen calibration. My experience building OpenHardware devices really pushed me to make the LVFS free-for-all, on the logic that I wouldn’t have been able to justify even a $100/year subscription. Certainly making the service free in all respects meant that it was almost risk-free for companies to test the service.

Now the LVFS analyses uploaded firmware for security problems, keeps millions of devices up to date, and also helps governments buy secure hardware using initiatives like the upcoming Host Security ID that I’ll talk more about in future blog posts. How many devices we’ve updated is impossible to know exactly as many large companies and departments mirror the entire LVFS; we just know it’s at least 20 million. In reality it’s probably a single digit multiple of that, although there’s no real way of knowing. We know 1.5 million people have sent the optional “it works for me” report we ask from CLI users, and given the CLI downloads account for ~1% of all downloads it could be a lot higher than 20 million.

A huge number of devices are supported on the LVFS now. There are currently 2393 different public firmwares uploaded by 1401 users from 106 different vendors, using 39 different protocols to update hardware. We’ve run 40,378 automated tests on those files, and extracted 1,170,543 file volume objects which can be scanned by YARA. All impressive numbers I’m sure you’ll agree.

There is one specific person I would like to thank first, my co-maintainer for both projects: Mario Limonciello who is a Senior Principal Software Development Engineer at Dell. Mario has reviewed thousands of my patches over the years, and contributed hundreds himself. I really appreciate his trust and also his confidence to tell me the half-baked and incomplete thing I’m proposing is actually insane. Together we’ve created an architecture that’s easy to maintain with a clean and modern design. Dell were also the first major laptop OEM to tell their suppliers “you need to ship firmware updates using the LVFS” and so did a lot of the initial plumbing work. We re-used most of the initial plugins when other OEMs decided they’d like to join the initiative later.

Peter Jones is another talented member of our team at Red Hat and wrote a lot of the low level UEFI code we’ve used millions of times. Peter has to understand all the crazy broken things that firmware vendors decide to do, and is responsible for most of the EFI code in fwupd. Over the years fwupd has absorbed two of his projects, fwupdate and most recently dbxtool. Without Peter there would have been no UpdateCapsule support, and that’s about half the updates on the LVFS.

Also notable to mention here is Logitech. They’ve shipped a ton of firmware (literally, millions) for their Unifying hardware and were also early adopters of the LVFS and fwupd. Nestor Lopez Casado, many thanks for all your help over the years and I’m glad MouseJack made all this a requirement :)

I’d also like to thank Lenovo; not a specific person, as Lenovo is split up into ThinkPad, ThinkStation and ThinkCenter groups and in each the engineers would probably like to remain anonymous. Lenovo as a combined group is shipping a huge amount of firmware now via the LVFS, and most of the Lenovo supply chain is already wired up to supporting the LVFS. A lot of the ODMs for Lenovo have had to actually install Fedora and learn how to program with GLib C to create a fwupd plugins to support laptop models that are not even on the shelves yet. Training up dozens of people in “how to write a fwupd plugin and deal with a grumpy maintainer” took a lot of time, but now we have companies building custom silicon submitting ready-to-roll plugins, fixes and even enhancements to the GUI tools.

The LVFS isn’t just a way for OEMs to distribute firmware, like a shared FTP site. The LVFS plumbs-itself into the ODM and ISV relationships, so we can get a pipeline right from the firmware author, all the way to the end user. As ODMs such as Wistron and Foxconn use the LVFS for one OEM, it’s very simple for them to also support other OEMs. The feedback loop from vendors to users and back to vendors again has been invaluable when debugging problems with specific firmware releases.

More recently various groups at Google have also been pushing suppliers of the Chromebook ecosystem to use fwupd and the LVFS. I’ve been told there’s now a “fwupd group” inside Google and I know that there are more than a few different models that will only be updatable using fwupd. Google are also using some of the consulting companies familiar with LVFS and fwupd so that it’s not just me explaining how this works over and over to various different small ODMs. I think in the next year this consulting side will explode and help grow the ecosystem even further.

I’d love to list all the OEMs, ODMs, ISVs and consultants that have helped over the last 5 years, but this blog entry would be even larger than it already is. Just know that I appreciate your support, help and guidance.

Talk of growing the ecosystem, the Linux Foundation are taking over the actual site maintenance; I’m not a sysadmin, and Terraform and Docker still scare me. This Christmas we’ll move the little VM I have running in Amsterdam to a proper scalable architecture with 24/7 support. This will let us provide some kind of uptime guarantee and also means I don’t have to worry about applying a security update when I’m on holiday without internet access. The Linux Foundation have been paying the entire CDN cost for the last few years, and for that my wallet is truly grateful.

Finally, I must also thank Red Hat for letting me work on this stuff. Over the last few years fwupd has gone from my “20%” hobby to almost taking over all my developer time. Red Hat doesn’t get enough credit for all the essential plumbing they tirelessly do, and without them paying my salary every month there is no way this kind of free-to-upload and free-to-download service could exist. I firmly believe that fwupd is mutually beneficial to all the Red Hat partners like Intel, Dell and Lenovo – both so that more hardware gets purchased from OEMs, and also so that customers running Fedora and RHEL have up to date firmware.

At a conference last year, I presented a talk where the penultimate slide was “the LVFS is just a website that runs cron jobs” and I had someone I respect come up to me afterwards and tell me something in a stern voice I’ll remember forever: “You didn’t just create a website – you changed an industry!

Lets look forward to the next 20 million updates.

Updating Secure Boot dbx with fwupd and the LVFS

This is one of those blog posts which is going to escalate quickly; in my effort to make it understandable I might simplify some of the theory, so if you know all this stuff already please scroll down a few paragraphs and try not to be pedantic.

UEFI Secure Boot is protection technology that is designed to protect a system against malicious code being executed early in the boot process. It defines a way to “lock down” the platform so only binaries signed with a specific key will run. The certificate authority is managed by Microsoft, which is unimportant generally, but was hugely important before Microsoft agreed to sign loaders like shim as it meant hardware wouldn’t run Linux. SecureBoot isn’t evil by itself, and having SecureBoot turned on protects your hardware from real world attacks and in theory secures most of the boot process to where the Linux kernel takes over. In the case where you don’t love or trust Microsoft you can configure another certificate authority to manage the trust somewhere else, but that’s a blog post for another day.

Anyway — I alluded that Microsoft has a special process for signing our Linux binaries so we could still keep running Linux on hardware that can’t turn off SecureBoot for one reason or another. What happened with the BootHole set of CVEs was that a researcher from Eclypsium found a nasty bug in the GRUB project (of which the binary was signed by Microsoft) which allowed complete circumvention of the SecureBoot architecture. This would mean a bootkit that previously would only work with SB turned off (which is fairly uncommon, as Microsoft forces OEMs to turn it on by default) now could be easily deployed onto hardware with SB enabled using an old version of shim packaged in the bootkit.

Luckily, this scenario was planned for by the UEFI people, and as well as having a “binaries must be signed by this certificate” functionality we also have a “the binary must not have this checksum” back up plan. Microsoft recently invoked the back-up-plan and added quite a few checksums of binaries like shim and grub to the latest dbx update. They’ve done this three times since 2010, but only traditionally for things like the Symantec recovery binary which most people won’t have installed. The signed “blocklist of checksums” is called the dbx internally, and can only be updated and can never be downgraded for obvious reasons.

If you actually deploy the current UEFI Revocation List (a.k.a dbx) on your Linux box right now using something like dbxtool it’ll most probably apply fine, and then when you reboot you’ll get a nice red screen and a SECURE BOOT VIOLATION message, and then for most people the computer will be a useless brick. The grub and shim installed into /boot/efi by your OS is now being blocked, and so nothing works. You can’t even boot a Linux installer to reinstall as the grub on the live media is going to be blocked too.

What we used to do was have a dbxtool.service that just applied the latest dbx (if you were not using a LiveCD) and then just hope it all still boots. This mostly worked, as the dbx contained checksums of things you were not likely to be using, rather than things you are most likely to have. You might also be thinking this also isn’t an awesome way to deploy a single-use binary to millions of computers. It’s like shipping firmware updates in an rpm. Hmm. That gives me an idea.

What we could do is have a fwupd plugin that reads the current dbx and creates a device:

│   
└─UEFI dbx:
      Device ID:           362301da643102b9f38477387e2193e57abaa590
      Summary:             UEFI Revocation Database
      Current version:     77
      Minimum Version:     77
      Install Duration:    1 second
      GUIDs:               5971a208-da00-5fce-b5f5-1234342f9cf7 ← UEFI\CRT_A9087D1044AD18F7A94916D284CBC01827CF23CD8F60B79072C9CAA1FEF4D649&ARCH_X64
                           f8ba2887-9411-5c36-9cee-88995bb39731 ← UEFI\CRT_A1117F516A32CEFCBA3F2D1ACE10A87972FD6BBE8FE0D0B996E09E65D802A503&ARCH_X64
      Device Flags:        • Internal device
                           • Updatable
                           • Supported on remote server
                           • Needs a reboot after installation

Then we could push the dbx updates onto the LVFS so that they get downloaded as required rather than stored into /usr for all eternity. We could put all the checks into fwupd to verify that the user doesn’t have any blocked binaries into the ESP (as fwupd already knows how to mount the ESPs to deploy UEFI capsules) so that we don’t accidentally brick any systems:

And we could wire this up to the GUI so that it’s super easy to use, and makes it a one-click install to deploy locally or scriptable using the CLI:

If you install fwupd and gnome-software from git master, this is exactly what you can do right now. To test this on your non-production system, you can add something like this to /etc/fwupd/remotes.d/dbx.conf:

[fwupd Remote]
Enabled=true
Title=UEFI dbx
Keyring=gpg
MetadataURI=https://fwupd.org/downloads/firmware-1d780afd8789afc6bccff638e5e8376604ba082aa32dc06db71d6768824a1efa.xml.gz
ReportURI=https://fwupd.org/lvfs/firmware/report
OrderBefore=lvfs,fwupd

Then fwupdmgr refresh gets you the metadata and fwupdmgr update applies the update only if it’s safe to do so. Again: Do not do this on a system you don’t have backups for.

I’m going to test this myself on a few more machines and then once the shim and grub builds hit Fedora stable we can think about packaging up the new fwupd so that we can deploy the dbx configuration using the LVFS easily. Of course, nobody is taking away doing it manually with dbxtool, but now it’s integrated into the system and easy for the user to deploy. Comments and corrections most welcome.

Using Red Hat Satellite with the LVFS

A months weeks ago I alluded that you could run the LVFS in an offline mode, where updates could be synced to a remote location and installed on desktops and servers in a corporate setting without internet access. A few big companies asked for more details, and so we created some official documentation which should help. It works, but you need to script things manually and set up your system in a custom way.

For those companies using Red Hat Satellite there’s now an even easier way. Marc Richter has created a public Red Hat knowledge base article on how to configure Satellite 6 so that firmware updates can be deployed on client systems without access to the public internet. This is something that’s been requested by Red Hat customers for some time, and I really appreciate all the research and debugging Marc had to do so that this kind of content could be written. Feel free to share the link to this content as it’s available without a Red Hat subscription. Comments welcome!

Now firmware can depend on available client features

At the moment we just blindly assume the capabilities of the front-end client when installing firmware. We can somewhat work around this limitation by requiring a new enough fwupd daemon version, but the GUI client software may be much older than the fwupd version or just incomplete. If you maintain a text or graphical client that uses fwupd to deploy updates then there’s an additional API call I’d like you to start using so we can fix this limitation.

This would allow, for instance, the firmware to specify that it requires the client to be able to show a runtime detach image. This would not be set by a dumb command line tool using FwupdClient, but would be set by a GUI client that is capable of downloading a URL and showing a PNG to the user.

Clients that do not register features are assumed to be dumb and won’t be offered firmware that has a hard requirement on showing a post-install “you need to restart the hardware manually” image and caption. The three known actions you can register for client feature support are can-report, detach-action and the recently added update-action. See this commit for more details about what each feature actually means.

If you’re using libfwupd then it’s a simple call to fwupd_client_set_feature_flags() otherwise you’ll have to call the SetFeatureFlags() on the main D-Bus interface before requesting the list of updates. Simple!

Converting to encrypted swap

I’m working on a firmware platform security specification which we will announce soon. Most of the things we test are firmware protections the user cannot actually change, but there are some runtime checks we do to make sure we can actually trust the results from the kernel. For instance, if you load unknown random modules into the kernel (which means it becomes “tainted”) you can’t actually trust the values reported. Another basic sanity check we do is checking for encrypted swap space.

My Lenovo P50 was installed with Fedora 29ish, a long time ago, with encrypted /home and unencrypted swap. It’s been upgraded quite a few times and I’m not super keen on re-installing it now. I wanted to upgrade to encrypted swap so I could pass the same requirements that I’m going to be asking people to meet.

Please don’t just copy and paste the below, as you will have a different swap partition to me. If you choose the wrong partition you will either overwrite your data or your root partition, so be careful. Caveat emptor, and all that.

So, lets get started. Lets turn off the existing swap partition:

[root@localhost ~]# cat /proc/swaps
Filename				Type		Size	Used	Priority
/dev/nvme0n1p4                          partition	5962748	0	-2
[root@localhost ~]# swapoff /dev/nvme0n1p4

Lets overwrite the existing partition with zeros, as it might have data that we’d consider private:

dd if=/dev/zero of=/dev/nvme0n1p4 bs=102400

We then need to change /etc/fstab from

# Created by anaconda on Mon Dec  9 09:05:10 2019
...
UUID=97498951-0a49-4110-b838-dd90d02ea11f none                    swap    defaults        0 0
...

to

...
/dev/mapper/swap                          none                    swap    defaults        0 0    
...

We then need to append to /etc/crypttab:

swap /dev/nvme0n1p4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256

Reboot, and then cat /proc/swaps will show you using a dm device. Done!

Initial release of Jcat

Today I released the first official tarball of Jcat, version 0.1.0. I’ve started the process to get the package into Fedora as it will almost certainly be a hard requirement in the next major version of fwupd.

Since I announced Jcat a few weeks ago, I’ve had a lot of positive feedback about the general concept and, surprisingly, even one hardware vendors suggested they might start self-signing their firmware before uploading to the LVFS (which is great!). More LVFS announcements coming soon I promise…

The LVFS has been including Jcat files in archives and generating them for metadata for about three weeks now, and we’ve had no issues reported. Once the package is available in Fedora 32 I’ll merge the fwupd pull request to make it a hard dep. All you other distro package maintainers, please go do your packaging thing!

If anyone finds any oddities or weird behavior, please file an issue. I’m not expecting to make API breaks now, but will if we find a design bug. Most of the code is imported from fwupd, and so I’m pretty comfortable with the general design. Comments welcome.

Introducing Jcat

In 2015 I wrote a bit about Microsoft Catalog files and threatened to invent a new specification if nobody else stepped up to the plate. Microsoft have still not documented the format, and have seemingly broken the on-disk file format at least once, and there’s still no easy way we can write these files in Linux. Yesterday was a long day, and I think I have a prototype I want to share with the world now. Lets go back a bit first, what is a catalog file anyway?

A catalog file is a container format that can hold any number of signatures for external files. The data files themselves are not contained in the catalog, only the signatures. Detached signatures are something we’re familiar with in Linux, typically in the form of .asc files for GPG and .p7b files for PKCS-7 protocols. Detached files are something we also generate on the LVFS and consume in fwupd and this is the reason you can see more than just the README.txt, firmware.bin and firmware.metainfo.xml in the cabinet archive:

The bug we’re trying to fix was we wanted to sign the metainfo file too so we can be sure that it came from the LVFS, rather than been modified by anyone capable of extracting and creating a new cabinet archive. This would have added firmware.metainfo.xml.asc and firmware.metainfo.xml.p7b to the archive, and at some point you have to wonder just how many extensions we can nest before the world descends into chaos. It also meant that we would have to add an extra n_firmware_files + n_metainfo_files detached signatures to each archive for every extra signing mechanism we add.

The other problem with a single detached signature per engine per file is that only one entity can sign the firmware. At the moment it’s just the LVFS, but what if Dell wanted to sign the firmware with a detached signature too, saying “this firmware is 100% from Dell”? Would we have com.dell.firmware.metainfo.xml.p7b too? This all got very complicated and basically forced me to create a new project to make it all much simpler.

Jcat is a gzipped JSON file of detached signatures. Because it’s gzipped it’s easy to compress and decompress in basically any language, and because it’s JSON it’s dead simple to parse and generate in any framework. There is a little overhead of some metadata (e.g. signing ID, creation time, etc) and but it’s all the kind of thing you can just edit in vim if you needed to. There’s also support for storing binary stuff like DER certificates (base64 to the rescue…), but if possible I’d like it to be all readable in a text editor. The jcat command line tool can import existing detached signatures into the Jcat file, and can also verify the existing .jcat file against all the files in a directory or archive. You can include multiple signatures for the same file (using the AppStream ID as the key) and of course sign multiple files using all the cryptographic engines you need. There’s also rudimentary support for actually creating signatures in the jcat command line client too, although it’s WIP for the GNUTLS engine and completely missing for GPGME at the moment.

This new thing also lets us fix another glaring issue in fwupd. Some companies can’t use PKCS-7, and some can’t use GPG for equally bad and nonsensical reasons – at the moment you need to specify the remote keyring when enabling a remote as we need to know if we need to download the metadata.xml.gz.asc or the .p7b version. Using a .jcat file allows to to not care, and just download one detached thing that can be used no matter how you’ve compiled your system. By adding SHA-256 as an additional not-to-be-used-for-trust engine, Jcat also lets you verify the download of your metadata and cabinet files even when you don’t have GPG or PKCS-7 available, which I know at least one company does on an IOT project. Jcat allows us to move the scary cryptographic verification code out of fwupd and makes the update-your-firmware codebase easier to maintain without worrying about potential landmines.

I’ve got a wip/hughsie/jcat branch on fwupd, and the same for lvfs-website although none of it has had any kind of peer review. Feedback on the initial release of libjcat most welcome from anyone familiar with writing libraries with GIO and Glib.

I suck at naming, but Jcat is supposed to be “JSON Catalog”. My daughters also have lots of Jellycat toys scattered around the house too, and the name seemed not to be taken by any other projects so far. If anyone knows of a project already using the name, please let me know. Feedback very welcome.

Hunting UEFI Implants

Last week I spent 3 days training on how to detect UEFI firmware implants. The training was run by Alex Matrosov via Hardwear.io and was a comprehensive deep-dive into UEFI firmware internals so that we could hunt for known and unknown implants. I’d 100% recommend this kind of training, it was excelent. Although I understood the general concepts of the protection mechanisms like SMM, HP Sure Start and Intel BIOSGuard before doing the training, it was really good to understand how the technologies really worked, with real world examples of where hardware vendors were getting the implementation wrong – giving the bad guys full control of your hardware. The training was superb, and Alex used lots of hands-on lab sessions to avoid PowerPoint overload. My fellow students were a mixture of security professionals and employees from various government departments from all over the world. We talked, a lot.

My personal conclusion quite simply is that we’re failing as an industry. In the pursuit to reduce S3 resume time from 2s to 0.5s we introduce issues like the S3 bootscript vulnerability. With the goal to boot as quickly as possible, we only check the bare minimum certificate chain allowing additional malicious DXEs to be added to an image. OEMs are choosing inexpensive EC hardware from sketchy vendors that are acting as root of trust and also emulating hardware designed 30 years ago, whilst sharing the system SPI chip. By trying to re-use existing power management primitives like SMM as a security boundary the leaky abstractions fail us. Each layer in the security stack is assuming that the lower below it is implemented correctly, and so all it takes is one driver with SMM or CSME access to not check a memory address in a struct correctly and everything on top (e.g. BootGuard, ALSR, SELinux, etc) is broken. Coreboot isn’t the panacea here either as to get that to run you need to turn off various protections like BootGuard, and some techniques like Sure Start mean that Coreboot just isn’t a viable option. The industry seems invested into EDK2, for better or worse. This shouldn’t just be important to the few people just buying stuff from Purism – 10,000x laptops are being sold on Amazon for every laptop sold by vendors that care about this stuff.

Most of the easy-to-exploit issues are just bugs with IBV or ODM-provided code, some of which can be fixed with a firmware update. Worst still, if you allow your “assumed secure” laptop out of sight then all bets are off with security. About a quarter of people at the UEFI training had their “travel laptop” tampered with at some point – with screws missing after “customs inspections” or with tamper seals broken after leaving a laptop in a hotel room. You really don’t need to remove the screws to image a hard drive these days. But, lets back away from the state-sponsored attacker back to reality for a minute.

The brutal truth is that security costs money. Vendors have to choose between saving 10 cents on a bill-of-materials by sharing a SPI chip (so ~$10K over a single batch), or correctly implementing BIOSGuard. What I think the LVFS now needs to do is provide some easy-to-understand market information to people buying hardware. We already know a huge amount of information about the device from signed reports and from analyzing the firmware binaries. What we’re not doing very well is explaining it to the user in a way they can actually understand. I didn’t understand the nuances between BIOSGuard and BootGuard until a few days ago, and I’ve been doing this stuff for years.

What I propose we do is assign some core protections some weight, and then verify and document how each vendor is configuring each model. For instance, I might say that for my dads laptop any hardware “SEC1” and above is fine as he’s only using it for Facebook or YouTube and it needs to be inexpensive. For my personal laptop I would be happy to restrict my choice of models to anything SEC3 and above. If you’re working as a journalist under some corrupt government, or am a security researcher, only SEC4 and above would be suitable. The reality is that SEC4 is going to be several hundred dollars more expensive than some cheap imported no-name hardware that doesn’t even conform to SEC1.

Of course, we’ll need to expand the tests we do in fwupd to detect implementation errors, and to verify that the model that we’ve purchased does indeed match the SEC level advertised by the LVFS. I’m talking to a few different people on how to do this securely. What I do know is that it will involve a reboot to get some of the data that we can’t even get in kernel mode with SecureBoot turned on. The chipsec report gets us some of the way there, but it’s just too complicated for end users and won’t work with SB turned on.

My proposal would be as follows:

  • SEC1: SecureBoot, BIOS_WE, BLE, SMM_BWP, and updates on the LVFS, with no existing detectable SMM issues (like ThinkPwn for example)
  • SEC2: PRx set correctly, if not using BootGuard or BIOSGuard, with PCR0 attestation data
  • SEC3: BootGuard enabled, with the EC controller requiring signed images
  • SEC4: Intel BIOSGuard or HP SureStart
  • SEC5: Hardware attestation like Apple T2 or Google Titan

SEC1 is a really low bar. Anything not unsetting BIOS_WE can be flashed at runtime trivially. If you’re traveling with a laptop for work you really ought to be specifying at least a SEC3 level of protection.

Of course, some vendors might not care at all about security for some models. A “gaming laptop” with a flashing RGB keyboard backlight is really designed for playing games as fast as possible, and the fact that the BIOS is unlocked might be a good thing as then the user can flash a custom unsigned BIOS with a CounterStrike-themed vendor image. I don’t think we ought to make vendors feel guilty about not even hitting SEC1. Perhaps we could let the consumer vote with their wallet and make the ecosystem more secure. I’m not sold on the SECx name either, it’s not very catchy. I suck at naming stuff. Comments welcome.

Improving the security model of the LVFS

There are lots of layers of security in the LVFS and fwupd design, including restricted account modes, 2FA, and server side AppStream namespaces. The most powerful one is the so-called vendor-id that the vendors cannot assign themselves, and is assigned by me when creating the vendor account on the LVFS. The way this works is that all firmware from the vendor is tagged with a vendor-id string like USB:0x056A which in this case matches the USB consortium vendor assigned ID. Client side, the vendor-id from the signed metadata is checked against the physical device and the firmware is updated only if the ID matches. This ensures that malicious or careless users on the LVFS can never ship firmware updates for other vendors hardware. About 90% of the vendors on the LVFS are locked down with this mechanism.

Some vendors have to have IDs that they don’t actually own, a good example here is for a DFU device like the 8bitdo controllers. In runtime mode they use the USB-assigned 8bitdo VID, but in bootloader mode they use a generic VID which is assigned to the chip supplier as they are using the reference bootloader. This is obviously fine, and both vendor IDs are assigned to 8bitdo on the LVFS for this reason. Another example is where Lenovo is responsible for updating Lenovo-specific NVMe firmware, but where the NVMe vendor isn’t always Lenovo’s PCI ID.

Where this breaks down a little more is for hardware devices that don’t have a built-in assigned vendor mapping. There are three plugins which are causing minor headaches:

  • Redfish — there’s seemingly no PCI vendor code for the enumerated devices themselves
  • ATA — the ATA-ATAPI-5 specification bizarrely makes no mention of any kind of vendor ID in the IDENTIFY block
  • UEFI — the ESRT table frustratingly just lists the version number and the GUID of devices, but no actual sysfs link to each

All the other plugins can be handled in a sane way, mostly automatically as the vast majority derive from either FuUsbDevice or FuUdevDevice.

As UEFI UpdateCapsule updates seem to be the 2nd most popular way to distribute firmware updates we probably ought to think of a sane way of limiting firmware updates to the existing BIOS vendor. We could query the DMI data, so that for instance Lenovo is only able to update Lenovo hardware — but we have to use a made-up pseudo-vendor-id of DMI:Lenovo. Maybe this isn’t so bad. Perhaps the vendor ID isn’t so useful with UEFI Update Capsule as the capsules themselves have to be signed by the firmware vendor before they’ll actually be run.

Anyway, to the point of this blog post: Until recently fwupd would refuse to apply the update if the metadata contained a vendor-id, but the device had not set one. This situation now might happen if for instance a vendor had to have no vendor-id because the device traditionally had no PCI or USB VID, and now in newer versions of fwupd the device would actually have a virtual ID, and so the vendor could be locked down on the LVFS. The fix here is to ignore the metadata vendor-id if there’s no device vendor-id, rather than failing to update.

Most people should be running fwupd 1.3.x, which is the latest and greatest branch of fwupd. I appreciate some LTS distros can’t rebase to a newer minor version, and so for old versions of fwupd I’ve backported the fix. These are the fixes you want if you’re running 0.9.x, 1.0.x, 1.1.x or 1.2.x.

I’ll make the vendor-id a hard requirement for all vendors in about 6 months time, so if you maintain a distro packaged version of fwupd you have that much time before some updates will stop working. If anyone has comments or concerns, please let me know.