Technical Blog of Richard Hughes

Lenovo ThinkPad and ThinkStation have already been using the LVFS for some time, with many models supported from each group. Now the first firmware for the ThinkCentre line of hardware has appeared. ThinkCentre machines are often found in the enterprise, often tucked neatly behind other hardware or under counter tops, working away for years without problems. With the LVFS support site administrators can now update firmware on machines either locally or using ssh. At the moment only the M625q model is listed as supported on the LVFS, but other models are in the pipeline and will appear when ready.

It’s been a good month for the LVFS, 6 new devices were added in the last month, and we celebrated the numerically significant 5 million firmware downloads. The move to the Linux Foundation is going well, and we’ll hopefully be moving the staging instance from a little VM to a proper cloud deployment, providing the scalability and uptime requirements we need for critical infrastructure like this. If all goes to plan the main instance will move after a few months of testing.

It’s probably no surprise to many of you that PackageKit has been in maintenance mode for quite some time. Although started over ten years ago (!) it’s not really had active maintenance since about 2014. Of course, I’ve still been merging PRs and have been slinging tarballs over the wall every few months, but nothing new was happening with the project, and I’ve worked on many other things since.

I think it’s useful for a little retrospective. PackageKit was conceived as an abstraction layer over about a dozen different package management frameworks. Initially it succeeded, with a lot of front ends UIs being written for the PackageKit API, making the Linux desktop a much nicer place for many years. Over the years, most package managers have withered and died, and for the desktop at least really only two remain, .rpm and .deb. The former being handled by the dnf PackageKit backend, and the latter by aptcc.

Canonical seems to be going all in on Snaps, and I don’t personally think of .deb files as first class citizens on Ubuntu any more – which is no bad thing. Snaps and Flatpaks are better than packages for desktop software in almost every way. Fedora is concentrating on Modularity and is joining with most of the other distros with a shared Flatpak and Flathub future and seems to be thriving because of it. If course, I’m missing out a lot of other distros, but from a statistics point of view they’re unfortunately not terribly relevant. Arch users are important, but they’re also installing primarily on the command line, not using an abstraction layer or GUI. Fedora is also marching towards an immutable base image using rpmostree, containers and flatpaks, and then PackageKit isn’t only not required, but doesn’t actually get installed at all in Fedora SilverBlue.

GNOME Software and the various KDE software centers already have an abstraction in the session; which they kind of have to to support per-user flatpak applications and per-user pet containers like Fedora Toolbox. I’ve also been talking to people in the Cockpit project and they’re in the same boat, and basically agree that having a shared system API to get the installed package list isn’t actually as useful as it used to be. Of course, we’ll need to support mutable systems for a long time (RHEL!) and so something has to provide a D-Bus interface to provide that. I’m not sure whether that should be dnfdaemon providing a PackageKit-compatible API, or it should just implement a super-simple interface that’s not using an API design from the last decade. At least from a gnome-software point of view it would just be one more plugin, like we have a plugin for Flatpak, a plugin for Snap, and a plugin for PackageKit.

Comments welcome.

The LVFS is a webservice designed to allow system OEMs and ODMs to upload firmware easily, and for it to be distributed securely to tens of millions of end users. For some people, this simply does not work for good business reasons:

• They don’t trust me, fwupd.org, GPG, certain OEMs or the CDN we use
• They don’t want thousands of computers on an internal network downloading all the files over and over again
• The internal secure network has no internet connectivity

For these cases there are a few different ways to keep your hardware updated, in order of simplicity:

Download just the files you need manually

Download the .cab files you found for your hardware and then install them on the target hardware via Ansible or Puppet using fwupdmgr install foo.cab — you can use fwupdmgr get-devices to get the existing firmware versions of all hardware. If someone wants to write the patch to add JSON/XML export to fwupdmgr that would be a very welcome thing indeed.

Download and deploy firmware as part of an immutable image

If you’re shipping an image, you can just dump the .cab files into a directory in the deployment along with something like /etc/fwupd/remotes.d/immutable.conf (only on fwupd >= 1.2.3):

[fwupd Remote]
Enabled=false
Title=Vendor (Automatic)
Keyring=none
MetadataURI=file:///usr/share/fwupd/remotes.d/vendor/firmware

Then once you disable the LVFS, running fwupdmgr or fwupdtool will use only the cabinet archives you deploy in your immutable image (or even from an .rpm for that matter). Of course, you’re deploying a larger image because you might have several firmware files included, but this is how Google ChromeOS is using fwupd.

Sync all the public firmware from the LVFS to a local directory

You can use Pulp to mirror the entire contents of the LVFS (not private or embargoed firmware, for obvious reasons). Create a repo pointing to PULP_MANIFEST and then sync that on a regular basis to download the metadata and firmware. The Pulp documentation can explain how to set all this up. Make sure the local files are available from a webserver in your private network using SSL.

Then, disable the LVFS by deleting/modifying lvfs.conf and then create a myprivateserver.conf file on the clients /etc/fwupd/remotes.d:

[fwupd Remote]
Enabled=true
Type=download
Keyring=gpg
MetadataURI=https://my.private.server/mirror/firmware.xml.gz
FirmwareBaseURI=https://my.private.server/mirror

Export a share to all clients

Again, use Pulp to create a big directory holding all the firmware (currently ~10GB), and keep it synced. This time create a NFS or Samba share and export it to clients. Map the folder on clients, and then create a myprivateshare.conf file in /etc/fwupd/remotes.d:

[fwupd Remote]
Enabled=false
Title=Vendor
Keyring=none
MetadataURI=file:///mnt/myprivateshare/fwupd/remotes.d/firmware.xml.gz
FirmwareBaseURI=file:///mnt/myprivateshare/fwupd/remotes.d

Create your own LVFS instance

The LVFS is a free software Python 3 Flask application and can be set up internally, or even externally for that matter. You have to configure much more this way, including things like generating your own GPG keys, uploading your own firmware and setting up users and groups on the server. Doing all this has a few advantages, namely:

• You can upload your own private firmware and QA it, only pushing it to stable when ready
• You don’t ship firmware which you didn’t upload
• You can control the staged deployment, e.g. only allowing the same update to be deployed to 1000 servers per day
• You can see failure reports from clients, to verify if the deployment is going well
• You can see nice graphs about how many updates are being deployed across your organisation

I’m hoping to make the satellite deployment LVFS use cases more concrete, and hopefully add some code to the LVFS to make this easier, although it’s not currently required for any Red Hat customer. Certainly a “setup wizard” would make setting up the LVFS much easier than obscure commands on the console.

Comments welcome.

As some of you have successfully guessed, HP Inc have been testing the LVFS for a few months now. There are now a few devices by HP available in the stable metadata and there are many more devices planned. If you’ve got a Z2, Z6, Z8, Z440, Z640 or Z840 system then you might want to check for an update in the GNOME Software updates panel or using fwupdmgr update in the terminal. There are quite a few updates with important fixes for various CVEs. I don’t know how many different models of HP hardware they are planning to support, or the order that they will be uploaded but I’m happy with progress.

With the addition of HP, the LVFS now has most major OEMs uploading firmware. There are a few exceptions, but even seemingly-unlikely companies like Microsoft are still interested in shipping firmware on the LVFS in the future. If you want to know more about joining the LVFS, please just send me an email.

A few vendors have been testing the NVMe firmware update code, and so far so good; soon we should have three more storage vendors moving firmware to stable. A couple of vendors also wanted to use the hdparm binary to update SATA hardware that’s not using the NVMe specification. A quick recap of the difference:

• NVMe: Faster, more expensive controller, one cut-out in the M.2 PCB header
• SATA: Slower, less expensive to implement, standard SATA or PATA connector, or two cut-outs in the M.2 PCB header

I’ve just merged a plugin developed with the donation of hardware and support of Star Labs. Any ATA-compatible drive (even DVD drives) supporting ATA_OP_DOWNLOAD_MICROCODE should be updatable using this new plugin, but you need to verify the TransferMode (e.g. 0x3, 0x7 or 0xe) before attempting an update to prevent data loss. Rather than calling into hdparm and screenscraping the output, we actually set up the sg_io_hdr_t structure and CDB buffer in the fwupd plugin to ensure it always works reliably without any additional dependencies. We only use two ATA commands and we can share a lot of the infrastructure with other plugins. For nearly all protocols, on nearly all devices, updating firmware is really a very similar affair.

There should soon be firmware on the LVFS that updates the StarDrive in the Star Lite laptop. I opened up the Star Lite today to swap the M.2 SSD to one with an old version and was amazed to find that it’s 80% battery inside; it reminded me of the inside of an iPad. Really impressive engineering considering the performance.

If any vendor is interested in deploying updates on the LVFS using the new ata plugin please let me know. Comments welcome.

I’ve just deployed two security enhancements to the LVFS. It’s important to note that is is proactive in response to suggestions from OEMs, and there has not been any security issue with the service.

• All passwords will be upgraded to a modern PBKDF2, in our case using SHA256. By logging in to the LVFS your password is automatically upgraded and no manual action is required. Any user accounts that have not been used by this time next year will be sent an email to remind them.
• Local users can now optionally secure their accounts using two factor authentication, in our case using OTP. Users can opt-in to 2FA in the usual “Profile” menu once logged in to the LVFS. In the profile section you can also test your OTP PIN before enabling it for your account. Two factor authentication is considered a very good way of securing your user account and is a very good idea for administrator, manager and QA user access levels. Although 2FA isn’t required for all account types at the moment, in the future we might tighten the security policy a little bit when we know it’s all working for everybody.

As a consequence of these changes, the login dialog for the LVFS now looks a little different. All the same buttons are there (forgot password etc) but now the login process is a 2-step process rather than a single process. For vendors using OAuth, nothing much changes, and if required 2FA should be enabled by your domain administrator rather than enabled on the LVFS. If anyone has any problems, please let me know.

A few weeks ago Sean from Star Labs asked me to start the process of joining the LVFS. Star Labs is a smaller Linux-friendly OEM based just outside London not far from where I grew up. We identified three types of firmware we could update, the system firmware, the EC firmware and the SSD firmware. The first should soon be possible with the recent pledge of capsule support from AMI, and we’ve got a binary for testing now. The EC firmware will need further work, although now I can get the IT8987 into (and out of) programming mode. The SSD firmware needed a fix to fwupd (to work around a controller quirk), but with the soon-to-be released fwupd it can already be updated:

Sean also shipped me some loaner hardware that could be recovered using manufacturing tools if I broke it, which makes testing the ITE EC flashing possible. The IT89 chip is subtly different to the IT87 chip in other hardware like the Clevo reference designs, but this actually makes it easier to support as there are fewer legacy modes. This will be a blog post all of it’s own.

In playing with hardware intermittently for a few weeks, I’ve got a pretty good feel for the “Lap Top” and “Star Lite” models. There is a lot to like, the aluminium cases feel both solid and tactile (like an XPS 13) and it feels really “premium” unlike the Clevo reference hardware. Star Labs doesn’t use the Clevo platform any more, which allows it to take some other bolder system design choices. Some things I love: the LED IPS screen, USB-C charging, the trackpad and keyboard. The custom keyboard design is actually a delight to use; I actually prefer it to my Lenovo P50 and XPS 13 for key-placement and key-travel. The touchpad seems responsive, and the virtual buttons work well unlike some of the other touchpads from other Linux-friendly OEMs. The battery life seems superb, although I’ve not really done enough discharge→charge→discharge cycles to be able to measure it properly. The front-facing camera is at the top of the bezel where it belongs, which is something the XPS 13 has only just fixed in the latest models. Nobody needs to see up my nose.

There are a few things that could be improved with the hardware in my humble opinion: The shiny bezel around the touchpad is somewhat distracting on an otherwise beautifully matte chassis. There is also only a microSD slot, when all my camera cards are full sized. The RAM is soldered in, and so can’t be upgraded in the future, and the case screws are not “captive” like the new Lenovos. It also doesn’t seem to have a ThunderBolt interface which might matter if you want to use this thing docked with a bazillion things plugged in. Some of these are probably cost choices, the Lap Top is significantly cheaper than the XPS 13 developer edition I keep comparing it against in my head.

I was also curious to try the vendor-supplied customized Ubuntu install which was supplied with it. It just worked, in every way, and for those installing other operating systems like Fedora or Arch all the different distros have been pre-tested with extra notes – a really nice touch. This is where Star Labs really shine, these guys really care about Linux and it shows. I’ve been impressed with the Lab Top and I’ll be sad to return it when all the hardware is supported by fwupd and firmware releases are available on the LVFS.

So, if you’re using Star Drive hardware already then upgrade fwupd to the latest development version, enable the LVFS testing remote using fwupdmgr enable-remote lvfs-testing and tell us how the process goes. For technical reasons you need to power down the machine and power it back up rather than just doing a “warm” reboot. In a few weeks we’ll do a proper fwupd release and push the firmware to stable.

Just like AMI, Phoenix is a huge firmware vendor, providing the firmware for millions of machines. If you’re using a ThinkPad right now, you’re most probably using Phoenix code in your mainboard firmware. Phoenix have been working with Lenovo and their ODMs on LVFS support for a while, fixing all the niggles that was stopping the capsule from working with the loader used by Linux. Phoenix can help customers build deliverables for the LVFS that use UX capsule support to make flashing beautiful, although it’s up to the OEM if that’s used or not.

It might seem slightly odd for me to be working with the firmware suppliers, rather than just OEMs, but I’m actually just doing both in parallel. From my point of view, both of the biggest firmware suppliers now understand the LVFS, and provide standards-compliant capsules by default. This should hopefully mean smaller Linux-specific OEMs like Tuxedo and Star Labs might be able to get signed UEFI capsules, rather than just getting a ROM file and an unsigned loader.

We’re still waiting for the last remaining huge OEM, but fingers crossed that should be any day now.

When fwupd writes firmware to devices, it often writes it, then does a verify pass. This is to read back the firmware to check that it was written correctly. For some devices we can do one better, and read the firmware hash and compare it against a previously cached value, or match it against the version published by the LVFS. This means we can detect some unintentional corruption or malicious firmware running on devices, on the assumption that the bad firmware isn’t just faking the requested checksum. Still, better than nothing.

Any processor better than the most basic PIC or Arduino (e.g. even a tiny $5 ARM core) is capable of doing public/private key firmware signing. This would use standard crypto using X.509 keys or GPG to ensure the device only runs signed firmware. This protects against both accidental bitflips and also naughty behaviour, and is unofficial industry recommended practice for firmware updates. Older generations of the Logitech Unifying hardware were unsigned, and this made the MouseJack hack almost trivial to deploy on an unmodified dongle. Newer Unifying hardware requires a firmware image signed by Logitech, which makes deploying unofficial or modified firmware almost impossible.

There is a snag with UEFI capsule updates, which is how you probably applied your last “BIOS” firmware update. Although the firmware capsule is signed by the OEM or ODM, we can’t reliably read the SPI EEPROM from userspace. It’s fair to say flashrom does work on some older hardware but it also likes disabling keyboard controllers and making the machine reboot when probing hardware. We can get a hash of the firmware, or rather, a hash derived from the firmware image with other firmware-related things added for good measure. This is helpfully stored in the TPM chip, which most modern laptops have installed.

Although the SecureBoot process cares about the higher PCR values to check all manners of userspace, we only care about the index zero of this register, so called PCR0. If you change your firmware, for any reason, the PCR0 will change. There is one PCR0 checksum (or a number slightly higher than one, for reasons) on all hardware of a given SKU. If you somehow turn the requirement for the hardware signing key off on your machine (e.g. a newly found security issue), or your firmware is flashed using another method than UpdateCapsule (e.g. DediProg) then you can basically flash anything. This would be unlikely, but really bad.

If we include the PCR0 in the vendor-supplied firmware.metainfo.xml file, or set it in the admin console of the LVFS then we can verify that the firmware we’re running right now is the firmware the ODM or OEM uploaded. This means you can have firmware 100% verified, where you’re sure that the firmware version that was uploaded by the vendor is running on your machine right now. This is good.

As an incentive for vendors to support signing there are now “green ticks” and “red crosses” icons for each firmware on the LVFS. All green ticks mean the firmware was uploaded to the LVFS by the OEM or authorized ODM on behalf of the OEM, the firmware is signed using strong encryption, and we can do secure attestation for verification.

Obviously some protocols can’t do everything properly (e.g. ColorHug, even symmetric crypto isn’t good) but that’s okay. It’s still more secure than flashing a random binary from an FTP site, which is what most people were doing before. Not upstream yet, and not quite finished, so comments welcome.

American Megatrends Inc. may not be a company you’ve heard of, unless perhaps you like reading early-boot BIOS messages. AMI is the world’s largest BIOS firmware vendor, supplying firmware and tools to customers such as Asus, Clevo, Intel, AMD and many others. If you’ve heard of a vendor using Aptio for firmware updates, that means it’s from them. AMI has been testing the LVFS, UpdateCapsule and fwupd for a few months and is now fully compatible. They are updating their whitepapers for customers explaining the process of generating a capsule, using the ESRT, and generating deliverables for the LVFS.

This means “LVFS Support” becomes a first class citizen alongside Windows Update for the motherboard manufacturers. This should trickle down to the resellers, so vendors using Clevo motherboards like Tuxedo get LVFS support almost for free. This will take a bit of time to trickle down to the smaller OEMs.

Also, expect another large vendor announcement soon. It’s the one quite a few people have been waiting for.