Initial release of Jcat

Today I released the first official tarball of Jcat, version 0.1.0. I’ve started the process to get the package into Fedora as it will almost certainly be a hard requirement in the next major version of fwupd.

Since I announced Jcat a few weeks ago, I’ve had a lot of positive feedback about the general concept and, surprisingly, even one hardware vendors suggested they might start self-signing their firmware before uploading to the LVFS (which is great!). More LVFS announcements coming soon I promise…

The LVFS has been including Jcat files in archives and generating them for metadata for about three weeks now, and we’ve had no issues reported. Once the package is available in Fedora 32 I’ll merge the fwupd pull request to make it a hard dep. All you other distro package maintainers, please go do your packaging thing!

If anyone finds any oddities or weird behavior, please file an issue. I’m not expecting to make API breaks now, but will if we find a design bug. Most of the code is imported from fwupd, and so I’m pretty comfortable with the general design. Comments welcome.

Introducing Jcat

In 2015 I wrote a bit about Microsoft Catalog files and threatened to invent a new specification if nobody else stepped up to the plate. Microsoft have still not documented the format, and have seemingly broken the on-disk file format at least once, and there’s still no easy way we can write these files in Linux. Yesterday was a long day, and I think I have a prototype I want to share with the world now. Lets go back a bit first, what is a catalog file anyway?

A catalog file is a container format that can hold any number of signatures for external files. The data files themselves are not contained in the catalog, only the signatures. Detached signatures are something we’re familiar with in Linux, typically in the form of .asc files for GPG and .p7b files for PKCS-7 protocols. Detached files are something we also generate on the LVFS and consume in fwupd and this is the reason you can see more than just the README.txt, firmware.bin and firmware.metainfo.xml in the cabinet archive:

The bug we’re trying to fix was we wanted to sign the metainfo file too so we can be sure that it came from the LVFS, rather than been modified by anyone capable of extracting and creating a new cabinet archive. This would have added firmware.metainfo.xml.asc and firmware.metainfo.xml.p7b to the archive, and at some point you have to wonder just how many extensions we can nest before the world descends into chaos. It also meant that we would have to add an extra n_firmware_files + n_metainfo_files detached signatures to each archive for every extra signing mechanism we add.

The other problem with a single detached signature per engine per file is that only one entity can sign the firmware. At the moment it’s just the LVFS, but what if Dell wanted to sign the firmware with a detached signature too, saying “this firmware is 100% from Dell”? Would we have com.dell.firmware.metainfo.xml.p7b too? This all got very complicated and basically forced me to create a new project to make it all much simpler.

Jcat is a gzipped JSON file of detached signatures. Because it’s gzipped it’s easy to compress and decompress in basically any language, and because it’s JSON it’s dead simple to parse and generate in any framework. There is a little overhead of some metadata (e.g. signing ID, creation time, etc) and but it’s all the kind of thing you can just edit in vim if you needed to. There’s also support for storing binary stuff like DER certificates (base64 to the rescue…), but if possible I’d like it to be all readable in a text editor. The jcat command line tool can import existing detached signatures into the Jcat file, and can also verify the existing .jcat file against all the files in a directory or archive. You can include multiple signatures for the same file (using the AppStream ID as the key) and of course sign multiple files using all the cryptographic engines you need. There’s also rudimentary support for actually creating signatures in the jcat command line client too, although it’s WIP for the GNUTLS engine and completely missing for GPGME at the moment.

This new thing also lets us fix another glaring issue in fwupd. Some companies can’t use PKCS-7, and some can’t use GPG for equally bad and nonsensical reasons – at the moment you need to specify the remote keyring when enabling a remote as we need to know if we need to download the metadata.xml.gz.asc or the .p7b version. Using a .jcat file allows to to not care, and just download one detached thing that can be used no matter how you’ve compiled your system. By adding SHA-256 as an additional not-to-be-used-for-trust engine, Jcat also lets you verify the download of your metadata and cabinet files even when you don’t have GPG or PKCS-7 available, which I know at least one company does on an IOT project. Jcat allows us to move the scary cryptographic verification code out of fwupd and makes the update-your-firmware codebase easier to maintain without worrying about potential landmines.

I’ve got a wip/hughsie/jcat branch on fwupd, and the same for lvfs-website although none of it has had any kind of peer review. Feedback on the initial release of libjcat most welcome from anyone familiar with writing libraries with GIO and Glib.

I suck at naming, but Jcat is supposed to be “JSON Catalog”. My daughters also have lots of Jellycat toys scattered around the house too, and the name seemed not to be taken by any other projects so far. If anyone knows of a project already using the name, please let me know. Feedback very welcome.

Hunting UEFI Implants

Last week I spent 3 days training on how to detect UEFI firmware implants. The training was run by Alex Matrosov via Hardwear.io and was a comprehensive deep-dive into UEFI firmware internals so that we could hunt for known and unknown implants. I’d 100% recommend this kind of training, it was excelent. Although I understood the general concepts of the protection mechanisms like SMM, HP Sure Start and Intel BIOSGuard before doing the training, it was really good to understand how the technologies really worked, with real world examples of where hardware vendors were getting the implementation wrong – giving the bad guys full control of your hardware. The training was superb, and Alex used lots of hands-on lab sessions to avoid PowerPoint overload. My fellow students were a mixture of security professionals and employees from various government departments from all over the world. We talked, a lot.

My personal conclusion quite simply is that we’re failing as an industry. In the pursuit to reduce S3 resume time from 2s to 0.5s we introduce issues like the S3 bootscript vulnerability. With the goal to boot as quickly as possible, we only check the bare minimum certificate chain allowing additional malicious DXEs to be added to an image. OEMs are choosing inexpensive EC hardware from sketchy vendors that are acting as root of trust and also emulating hardware designed 30 years ago, whilst sharing the system SPI chip. By trying to re-use existing power management primitives like SMM as a security boundary the leaky abstractions fail us. Each layer in the security stack is assuming that the lower below it is implemented correctly, and so all it takes is one driver with SMM or CSME access to not check a memory address in a struct correctly and everything on top (e.g. BootGuard, ALSR, SELinux, etc) is broken. Coreboot isn’t the panacea here either as to get that to run you need to turn off various protections like BootGuard, and some techniques like Sure Start mean that Coreboot just isn’t a viable option. The industry seems invested into EDK2, for better or worse. This shouldn’t just be important to the few people just buying stuff from Purism – 10,000x laptops are being sold on Amazon for every laptop sold by vendors that care about this stuff.

Most of the easy-to-exploit issues are just bugs with IBV or ODM-provided code, some of which can be fixed with a firmware update. Worst still, if you allow your “assumed secure” laptop out of sight then all bets are off with security. About a quarter of people at the UEFI training had their “travel laptop” tampered with at some point – with screws missing after “customs inspections” or with tamper seals broken after leaving a laptop in a hotel room. You really don’t need to remove the screws to image a hard drive these days. But, lets back away from the state-sponsored attacker back to reality for a minute.

The brutal truth is that security costs money. Vendors have to choose between saving 10 cents on a bill-of-materials by sharing a SPI chip (so ~$10K over a single batch), or correctly implementing BIOSGuard. What I think the LVFS now needs to do is provide some easy-to-understand market information to people buying hardware. We already know a huge amount of information about the device from signed reports and from analyzing the firmware binaries. What we’re not doing very well is explaining it to the user in a way they can actually understand. I didn’t understand the nuances between BIOSGuard and BootGuard until a few days ago, and I’ve been doing this stuff for years.

What I propose we do is assign some core protections some weight, and then verify and document how each vendor is configuring each model. For instance, I might say that for my dads laptop any hardware “SEC1” and above is fine as he’s only using it for Facebook or YouTube and it needs to be inexpensive. For my personal laptop I would be happy to restrict my choice of models to anything SEC3 and above. If you’re working as a journalist under some corrupt government, or am a security researcher, only SEC4 and above would be suitable. The reality is that SEC4 is going to be several hundred dollars more expensive than some cheap imported no-name hardware that doesn’t even conform to SEC1.

Of course, we’ll need to expand the tests we do in fwupd to detect implementation errors, and to verify that the model that we’ve purchased does indeed match the SEC level advertised by the LVFS. I’m talking to a few different people on how to do this securely. What I do know is that it will involve a reboot to get some of the data that we can’t even get in kernel mode with SecureBoot turned on. The chipsec report gets us some of the way there, but it’s just too complicated for end users and won’t work with SB turned on.

My proposal would be as follows:

  • SEC1: SecureBoot, BIOS_WE, BLE, SMM_BWP, and updates on the LVFS, with no existing detectable SMM issues (like ThinkPwn for example)
  • SEC2: PRx set correctly, if not using BootGuard or BIOSGuard, with PCR0 attestation data
  • SEC3: BootGuard enabled, with the EC controller requiring signed images
  • SEC4: Intel BIOSGuard or HP SureStart
  • SEC5: Hardware attestation like Apple T2 or Google Titan

SEC1 is a really low bar. Anything not unsetting BIOS_WE can be flashed at runtime trivially. If you’re traveling with a laptop for work you really ought to be specifying at least a SEC3 level of protection.

Of course, some vendors might not care at all about security for some models. A “gaming laptop” with a flashing RGB keyboard backlight is really designed for playing games as fast as possible, and the fact that the BIOS is unlocked might be a good thing as then the user can flash a custom unsigned BIOS with a CounterStrike-themed vendor image. I don’t think we ought to make vendors feel guilty about not even hitting SEC1. Perhaps we could let the consumer vote with their wallet and make the ecosystem more secure. I’m not sold on the SECx name either, it’s not very catchy. I suck at naming stuff. Comments welcome.

Improving the security model of the LVFS

There are lots of layers of security in the LVFS and fwupd design, including restricted account modes, 2FA, and server side AppStream namespaces. The most powerful one is the so-called vendor-id that the vendors cannot assign themselves, and is assigned by me when creating the vendor account on the LVFS. The way this works is that all firmware from the vendor is tagged with a vendor-id string like USB:0x056A which in this case matches the USB consortium vendor assigned ID. Client side, the vendor-id from the signed metadata is checked against the physical device and the firmware is updated only if the ID matches. This ensures that malicious or careless users on the LVFS can never ship firmware updates for other vendors hardware. About 90% of the vendors on the LVFS are locked down with this mechanism.

Some vendors have to have IDs that they don’t actually own, a good example here is for a DFU device like the 8bitdo controllers. In runtime mode they use the USB-assigned 8bitdo VID, but in bootloader mode they use a generic VID which is assigned to the chip supplier as they are using the reference bootloader. This is obviously fine, and both vendor IDs are assigned to 8bitdo on the LVFS for this reason. Another example is where Lenovo is responsible for updating Lenovo-specific NVMe firmware, but where the NVMe vendor isn’t always Lenovo’s PCI ID.

Where this breaks down a little more is for hardware devices that don’t have a built-in assigned vendor mapping. There are three plugins which are causing minor headaches:

  • Redfish — there’s seemingly no PCI vendor code for the enumerated devices themselves
  • ATA — the ATA-ATAPI-5 specification bizarrely makes no mention of any kind of vendor ID in the IDENTIFY block
  • UEFI — the ESRT table frustratingly just lists the version number and the GUID of devices, but no actual sysfs link to each

All the other plugins can be handled in a sane way, mostly automatically as the vast majority derive from either FuUsbDevice or FuUdevDevice.

As UEFI UpdateCapsule updates seem to be the 2nd most popular way to distribute firmware updates we probably ought to think of a sane way of limiting firmware updates to the existing BIOS vendor. We could query the DMI data, so that for instance Lenovo is only able to update Lenovo hardware — but we have to use a made-up pseudo-vendor-id of DMI:Lenovo. Maybe this isn’t so bad. Perhaps the vendor ID isn’t so useful with UEFI Update Capsule as the capsules themselves have to be signed by the firmware vendor before they’ll actually be run.

Anyway, to the point of this blog post: Until recently fwupd would refuse to apply the update if the metadata contained a vendor-id, but the device had not set one. This situation now might happen if for instance a vendor had to have no vendor-id because the device traditionally had no PCI or USB VID, and now in newer versions of fwupd the device would actually have a virtual ID, and so the vendor could be locked down on the LVFS. The fix here is to ignore the metadata vendor-id if there’s no device vendor-id, rather than failing to update.

Most people should be running fwupd 1.3.x, which is the latest and greatest branch of fwupd. I appreciate some LTS distros can’t rebase to a newer minor version, and so for old versions of fwupd I’ve backported the fix. These are the fixes you want if you’re running 0.9.x, 1.0.x, 1.1.x or 1.2.x.

I’ll make the vendor-id a hard requirement for all vendors in about 6 months time, so if you maintain a distro packaged version of fwupd you have that much time before some updates will stop working. If anyone has comments or concerns, please let me know.

Growing the fwupd ecosystem

Yesterday I wrote a blog about what hardware vendors need to provide so I can write them a fwupd plugin. A few people contacted me telling me that I should make it more generic, as I shouldn’t be the central point of failure in this whole ecosystem. The sensible thing, of course, is growing the “community” instead, and building up a set of (paid) consultants that can help the OEMs and ODMs, only getting me involved to review pull requests or for general advice. This would certainly reduce my current feeling of working at 100% and trying to avoid burnout.

As a first step, I’ve created an official page that will list any consulting companies that I feel are suitable to recommend for help with fwupd and the LVFS. The hardware vendors would love to throw money at this stuff, so they don’t have to care about upstream project release schedules and dealing with a gumpy maintainer like me. I’ve pinged the usual awesome people like Igalia, and hopefully more companies will be added to this list during the next few days.

If you do want your open-source consultancy to be added, please email me a two paragraph corporate-friendly blurb I can include on that new page, also with a link I can use for the “more details” button. If you’re someone I’ve not worked with before, you should be in a position to explain the difference between a capsule update and a DFU update, and be able to tell me what a version format is. I don’t want to be listing companies that don’t understand what fwupd actually is :)

Google and fwupd sitting in a tree

I’ve been told by several sources (but not by Google directly, heh) that from Christmas onwards the “Designed for ChromeBook” sticker requires hardware vendors to use fwupd rather than random non-free binaries. This does make a lot of sense for Google, as all the firmware flash tools I’ve seen the source for are often decades old, contain layer-on-layers of abstractions, have dubious input sanitisation and are quite horrible to use. Many are setuid, which doesn’t make me sleep well at night, and I suspect the security team at Google also. Most vendor binaries are built for the specific ODM hardware device, and all of them but one doesn’t use any kind of source control or formal review process.

The requirement from Google has caused mild panic among silicon suppliers and ODMs, as they’re having to actually interact with an open source upstream project and a slightly grumpy maintainer that wants to know lots of details about hardware that doesn’t implement one of the dozens of existing protocols that fwupd supports. These are companies that have never had to deal with working with “outside” people to develop software, and it probably comes as quite a shock to the system. To avoid repeating myself these are my basic rules when adding support for a device with a custom protocol in fwupd:

  • I can give you advice on how to write the plugin if you give me the specifications without signing an NDA, and/or the existing code under a LGPLv2+ license. From experience, we’ll probably not end up using any of your old code in fwupd but the error defines and function names might be similar, and I don’t anyone to get “tainted” from looking at non-free code, so it’s safest all round if we have some reference code marked with the right license that actually compiles on Fedora 31. Yes, I know asking the legal team about releasing previously-nonfree code with a GPLish licence is difficult.
  • If you are running Linux, and want our help to debug or test your new plugin, you need to be running Fedora 30 or 31. If you run Ubuntu you’ll need to use the snap version of fwupd, and I can’t help you with random Ubuntu questions or interactions between the snap version and the distro version. I know your customer might be running Debian Stable or Ubuntu LTS, but that’s not what I’m paid to support. If you do use Fedora 29+ or RHEL 7+ you can also use the nice COPR I provide with git snapshots of master.
  • Please reflect the topology of your device. If writes have to go through another interface, passthru or IC, please give us access to documentation about that device too. I’m fed up having to reverse engineer protocols from looking at the “wrong side” of the client source code. If the passthru is implemented by different vendor, they’ll need to work on the same terms as this.
  • If you want to design and write all of the plugin yourself, that’s awesome, but please follow the existing style and don’t try to wrap your existing code base with the fwupd plugin API. If your device has three logical children with different version numbers or firmware formats, we want to see three devices in fwupdmgr. If you want to restrict the child devices to a parent vendor, that’s fine, we now support that in fwupd and on the LVFS. If you’re adding custom InstanceIDs, these have to be documented in the README.md file.
  • If you’re using an nonstandard firmware format (as in, not DFU, Intel HEX or Motorola SREC) then you’ll need to write a firmware parser that’s going to be valgrind’ed and fuzzed. We will need all the header/footer documentation so we can verify the parser and add some small redistributable fuzz targets. If the blob is being passed to the hardware without parsing, you still might need to know the format of the header so that the plugin can do a sanity check that the firmware is suitable for the hardware, and that any internal CRC is actually correct. All the firmware parsers have to be paranoid and written defensively, because it’s me that looks bad on LWN if CVEs get issued.
  • If you want me to help with the plugin, I’m probably going to ask for test hardware, and two different versions of the firmware that can actually be flashed to the hardware you sent. A bare PCB is fine, but if you send me something please let me know so I can give you my personal address rather than have to collect it from a Red Hat office. If you send me hardware, ensure you also include a power supply that’s going to work in the UK, e.g. 240V. If you want it back, you’ll also need to provide me with UPS/DHL collection sticker.
  • You do need to think how to present your device version number. e.g. is 0x12345678 meant to be presented as “12.34.5678” or “18.52.86.120” – the LVFS really cares if this is correct, and users want to see the “same” version numbers as on the OEM web-page.
  • You also need to know if the device is fully functional during the update, or if it operates in a degraded or bootloader mode. We also need to know what happens if flashing fails, e.g. is the device a brick, or is there some kind of A/B partition that makes a flash failure harmless? If the device is a brick, how can it be recovered without an RMA?
  • After the update is complete fwupd need to “restart” the device so that the new firmware version can be verified, so there needs to be some kind of command the device understands – we can ask the user to reboot or re-plug the device if this is the only way to do this, although in 2019 we can really do better than that.
  • If you’re sharing a huge LGPLv2+ lump of code, we need access to someone who actually understands it, preferably the person that wrote it in the first place. Typically the code is uncommented and a recipe for a headache so being able to ask a human questions is invaluable. For this, either IRC, email or even just communicating via a shared Google doc (more common than you would believe…) is fine. I can’t discuss this stuff on Telegram, Hangouts or WhatsApp, sorry.
  • Once a plugin exists in fwupd and is upstream, we will expect pull requests to add either more VID/PIDs, #defines or to add variations to the protocol for new versions of the hardware. I’m going to be grumpy if I just get sent a random email with demands about backporting all the VID/PIDs to Debian stable. I have zero control on when Debian backports anything, and very little influence on when Ubuntu does a SRU. I have a lot of influence on when various Fedora releases get a new fwupd, and when RHEL gets backports for new hardware support.

Now, if all this makes me sound like a grumpy upstream maintainer then I apologize. I’m currently working with about half a dozen silicon suppliers who all failed some or all of the above bullets. I’m multiplexing myself with about a dozen companies right now, and supporting fwupd isn’t actually my entire job at Red Hat. I’m certainly not going to agree to “signing off a timetable” for each vendor as none of the vendors actually pay me to do anything…

Given interest in fwupd has exploded in the last year or so, I wanted to post something like this rather than have a 10-email back and forth about my expectations with each vendor. Some OEMs and even ODMs are now hiring developers with Linux experience, and I’m happy to work with them as fwupd becomes more important. I’ve already helped quite a few developers at random vendors get up to speed with fwupd and would be happy to help more. As the importance of fwupd and the LVFS grows more and more, vendors will need to hire developers who can build, extend and support their hardware. As fwupd grows, I’ll be asking vendors to do more of the work, as “get upstream to do it” doesn’t scale.

Synaptics CX Audio Support

A couple of weeks ago, Synaptics (who now own Conexant) sent me 22,000+ lines of LGPLv2+ licensed C++ that was capable of updating the firmware of all the CXxxxx audio devices that exist in various laptops and peripherals. Most of last week was spent reading the code, and refactoring it to be a CX audio plugin in fwupd. There were a few things I could do to reduce the code size considerably:

  • Use the abstractions shared with all the other plugins, e.g. SREC file format processing, data chunking and low level USB HID
  • Drop support for hardware families which are no longer supported and not likely to receive updates
  • Remove the layers of abstractions and the macros-of-macros-of-macros so common with a codebase age measured in decades
  • Use helper objects in GLib and GObject rather than having to create everything from scratch

So, after all that we got down to a 1377 line fwupd plugin which is a 16x code reduction. It’s broadly comparable in functionality to the 22,000 line code drop but only works in fwupd as a plugin rather than as a standalone updater. To add support for new hardware to the plugin all we have to do is add an entry to the quirk file, which tells us which CX family the specific USB VID/PID is using. The rest is auto-detected.

I can’t tell you the OEM or the hardware all this work is being driven by, but eagle-eyed readers will work it out :) In some cases you might see an extra device appear in fwupdmgr get-devices if you’re running the soon-to-be-released fwupd 1.3.2 and hopefully we can get firmware updates which use this new device on the LVFS some time this year.

GNOME Firmware 3.34.0 Release

This morning I tagged the newest fwupd release, 1.3.1. There are a lot of new things in this release and a whole lot of polishing, so I encourage you to read the release notes if this kind of thing interests you.

Anyway, to the point of this post. With the new fwupd 1.3.1 you can now build just the libfwupd library, which makes it easy to build GNOME Firmware (old name: gnome-firmware-updater) in Flathub. I tagged the first official release 3.34.0 to celebrate the recent GNOME release, and to indicate that it’s ready for use by end users. I guess it’s important to note this is just a random app hacked together by 3 engineers and not something lovelingly designed by the official design team. All UX mistakes are my own :)

GNOME Firmware is designed to be a not-installed-by-default power-user tool to investigate, upgrade, downgrade and re-install firmware.
GNOME Software will continue to be used for updates as before. Vendor helpdesks can ask users to install GNOME Firmware rather than getting them to look at command line output.

We need to polish up GNOME Firmware going forwards, and add the last few features we need. If this interests you, please send email and I’ll explain what needs doing. We also need translations, although that can perhaps wait until GNOME Firmware moves to GNOME proper, rather than just being a repo in my personal GitLab. If anyone does want to translate it before then, please open merge requests, and be sure to file issues if any of the strings are difficult to translate or ambigious. Please also file issues (or even better merge requests!) if it doesn’t build or work for you.

If you just want to try out a new application, it takes 10 seconds to install it from Flathub.