Year: 2019

Improving the security model of the LVFS

There are lots of layers of security in the LVFS and fwupd design, including restricted account modes, 2FA, and server side AppStream namespaces. The most powerful one is the so-called vendor-id that the vendors cannot assign themselves, and is assigned by me when creating the vendor account on the LVFS. The way this works is that all firmware from the vendor is tagged with a vendor-id string like USB:0x056A which in this case matches the USB consortium vendor assigned ID. Client side, the vendor-id from the signed metadata is checked against the physical device and the firmware is updated only if the ID matches. This ensures that malicious or careless users on the LVFS can never ship firmware updates for other vendors hardware. About 90% of the vendors on the LVFS are locked down with this mechanism.

Some vendors have to have IDs that they don’t actually own, a good example here is for a DFU device like the 8bitdo controllers. In runtime mode they use the USB-assigned 8bitdo VID, but in bootloader mode they use a generic VID which is assigned to the chip supplier as they are using the reference bootloader. This is obviously fine, and both vendor IDs are assigned to 8bitdo on the LVFS for this reason. Another example is where Lenovo is responsible for updating Lenovo-specific NVMe firmware, but where the NVMe vendor isn’t always Lenovo’s PCI ID.

Where this breaks down a little more is for hardware devices that don’t have a built-in assigned vendor mapping. There are three plugins which are causing minor headaches:

  • Redfish — there’s seemingly no PCI vendor code for the enumerated devices themselves
  • ATA — the ATA-ATAPI-5 specification bizarrely makes no mention of any kind of vendor ID in the IDENTIFY block
  • UEFI — the ESRT table frustratingly just lists the version number and the GUID of devices, but no actual sysfs link to each

All the other plugins can be handled in a sane way, mostly automatically as the vast majority derive from either FuUsbDevice or FuUdevDevice.

As UEFI UpdateCapsule updates seem to be the 2nd most popular way to distribute firmware updates we probably ought to think of a sane way of limiting firmware updates to the existing BIOS vendor. We could query the DMI data, so that for instance Lenovo is only able to update Lenovo hardware — but we have to use a made-up pseudo-vendor-id of DMI:Lenovo. Maybe this isn’t so bad. Perhaps the vendor ID isn’t so useful with UEFI Update Capsule as the capsules themselves have to be signed by the firmware vendor before they’ll actually be run.

Anyway, to the point of this blog post: Until recently fwupd would refuse to apply the update if the metadata contained a vendor-id, but the device had not set one. This situation now might happen if for instance a vendor had to have no vendor-id because the device traditionally had no PCI or USB VID, and now in newer versions of fwupd the device would actually have a virtual ID, and so the vendor could be locked down on the LVFS. The fix here is to ignore the metadata vendor-id if there’s no device vendor-id, rather than failing to update.

Most people should be running fwupd 1.3.x, which is the latest and greatest branch of fwupd. I appreciate some LTS distros can’t rebase to a newer minor version, and so for old versions of fwupd I’ve backported the fix. These are the fixes you want if you’re running 0.9.x, 1.0.x, 1.1.x or 1.2.x.

I’ll make the vendor-id a hard requirement for all vendors in about 6 months time, so if you maintain a distro packaged version of fwupd you have that much time before some updates will stop working. If anyone has comments or concerns, please let me know.

Growing the fwupd ecosystem

Yesterday I wrote a blog about what hardware vendors need to provide so I can write them a fwupd plugin. A few people contacted me telling me that I should make it more generic, as I shouldn’t be the central point of failure in this whole ecosystem. The sensible thing, of course, is growing the “community” instead, and building up a set of (paid) consultants that can help the OEMs and ODMs, only getting me involved to review pull requests or for general advice. This would certainly reduce my current feeling of working at 100% and trying to avoid burnout.

As a first step, I’ve created an official page that will list any consulting companies that I feel are suitable to recommend for help with fwupd and the LVFS. The hardware vendors would love to throw money at this stuff, so they don’t have to care about upstream project release schedules and dealing with a gumpy maintainer like me. I’ve pinged the usual awesome people like Igalia, and hopefully more companies will be added to this list during the next few days.

If you do want your open-source consultancy to be added, please email me a two paragraph corporate-friendly blurb I can include on that new page, also with a link I can use for the “more details” button. If you’re someone I’ve not worked with before, you should be in a position to explain the difference between a capsule update and a DFU update, and be able to tell me what a version format is. I don’t want to be listing companies that don’t understand what fwupd actually is :)

Google and fwupd sitting in a tree

I’ve been told by several sources (but not by Google directly, heh) that from Christmas onwards the “Designed for ChromeBook” sticker requires hardware vendors to use fwupd rather than random non-free binaries. This does make a lot of sense for Google, as all the firmware flash tools I’ve seen the source for are often decades old, contain layer-on-layers of abstractions, have dubious input sanitisation and are quite horrible to use. Many are setuid, which doesn’t make me sleep well at night, and I suspect the security team at Google also. Most vendor binaries are built for the specific ODM hardware device, and all of them but one doesn’t use any kind of source control or formal review process.

The requirement from Google has caused mild panic among silicon suppliers and ODMs, as they’re having to actually interact with an open source upstream project and a slightly grumpy maintainer that wants to know lots of details about hardware that doesn’t implement one of the dozens of existing protocols that fwupd supports. These are companies that have never had to deal with working with “outside” people to develop software, and it probably comes as quite a shock to the system. To avoid repeating myself these are my basic rules when adding support for a device with a custom protocol in fwupd:

  • I can give you advice on how to write the plugin if you give me the specifications without signing an NDA, and/or the existing code under a LGPLv2+ license. From experience, we’ll probably not end up using any of your old code in fwupd but the error defines and function names might be similar, and I don’t anyone to get “tainted” from looking at non-free code, so it’s safest all round if we have some reference code marked with the right license that actually compiles on Fedora 31. Yes, I know asking the legal team about releasing previously-nonfree code with a GPLish licence is difficult.
  • If you are running Linux, and want our help to debug or test your new plugin, you need to be running Fedora 30 or 31. If you run Ubuntu you’ll need to use the snap version of fwupd, and I can’t help you with random Ubuntu questions or interactions between the snap version and the distro version. I know your customer might be running Debian Stable or Ubuntu LTS, but that’s not what I’m paid to support. If you do use Fedora 29+ or RHEL 7+ you can also use the nice COPR I provide with git snapshots of master.
  • Please reflect the topology of your device. If writes have to go through another interface, passthru or IC, please give us access to documentation about that device too. I’m fed up having to reverse engineer protocols from looking at the “wrong side” of the client source code. If the passthru is implemented by different vendor, they’ll need to work on the same terms as this.
  • If you want to design and write all of the plugin yourself, that’s awesome, but please follow the existing style and don’t try to wrap your existing code base with the fwupd plugin API. If your device has three logical children with different version numbers or firmware formats, we want to see three devices in fwupdmgr. If you want to restrict the child devices to a parent vendor, that’s fine, we now support that in fwupd and on the LVFS. If you’re adding custom InstanceIDs, these have to be documented in the README.md file.
  • If you’re using an nonstandard firmware format (as in, not DFU, Intel HEX or Motorola SREC) then you’ll need to write a firmware parser that’s going to be valgrind’ed and fuzzed. We will need all the header/footer documentation so we can verify the parser and add some small redistributable fuzz targets. If the blob is being passed to the hardware without parsing, you still might need to know the format of the header so that the plugin can do a sanity check that the firmware is suitable for the hardware, and that any internal CRC is actually correct. All the firmware parsers have to be paranoid and written defensively, because it’s me that looks bad on LWN if CVEs get issued.
  • If you want me to help with the plugin, I’m probably going to ask for test hardware, and two different versions of the firmware that can actually be flashed to the hardware you sent. A bare PCB is fine, but if you send me something please let me know so I can give you my personal address rather than have to collect it from a Red Hat office. If you send me hardware, ensure you also include a power supply that’s going to work in the UK, e.g. 240V. If you want it back, you’ll also need to provide me with UPS/DHL collection sticker.
  • You do need to think how to present your device version number. e.g. is 0x12345678 meant to be presented as “12.34.5678” or “18.52.86.120” – the LVFS really cares if this is correct, and users want to see the “same” version numbers as on the OEM web-page.
  • You also need to know if the device is fully functional during the update, or if it operates in a degraded or bootloader mode. We also need to know what happens if flashing fails, e.g. is the device a brick, or is there some kind of A/B partition that makes a flash failure harmless? If the device is a brick, how can it be recovered without an RMA?
  • After the update is complete fwupd need to “restart” the device so that the new firmware version can be verified, so there needs to be some kind of command the device understands – we can ask the user to reboot or re-plug the device if this is the only way to do this, although in 2019 we can really do better than that.
  • If you’re sharing a huge LGPLv2+ lump of code, we need access to someone who actually understands it, preferably the person that wrote it in the first place. Typically the code is uncommented and a recipe for a headache so being able to ask a human questions is invaluable. For this, either IRC, email or even just communicating via a shared Google doc (more common than you would believe…) is fine. I can’t discuss this stuff on Telegram, Hangouts or WhatsApp, sorry.
  • Once a plugin exists in fwupd and is upstream, we will expect pull requests to add either more VID/PIDs, #defines or to add variations to the protocol for new versions of the hardware. I’m going to be grumpy if I just get sent a random email with demands about backporting all the VID/PIDs to Debian stable. I have zero control on when Debian backports anything, and very little influence on when Ubuntu does a SRU. I have a lot of influence on when various Fedora releases get a new fwupd, and when RHEL gets backports for new hardware support.

Now, if all this makes me sound like a grumpy upstream maintainer then I apologize. I’m currently working with about half a dozen silicon suppliers who all failed some or all of the above bullets. I’m multiplexing myself with about a dozen companies right now, and supporting fwupd isn’t actually my entire job at Red Hat. I’m certainly not going to agree to “signing off a timetable” for each vendor as none of the vendors actually pay me to do anything…

Given interest in fwupd has exploded in the last year or so, I wanted to post something like this rather than have a 10-email back and forth about my expectations with each vendor. Some OEMs and even ODMs are now hiring developers with Linux experience, and I’m happy to work with them as fwupd becomes more important. I’ve already helped quite a few developers at random vendors get up to speed with fwupd and would be happy to help more. As the importance of fwupd and the LVFS grows more and more, vendors will need to hire developers who can build, extend and support their hardware. As fwupd grows, I’ll be asking vendors to do more of the work, as “get upstream to do it” doesn’t scale.

Synaptics CX Audio Support

A couple of weeks ago, Synaptics (who now own Conexant) sent me 22,000+ lines of LGPLv2+ licensed C++ that was capable of updating the firmware of all the CXxxxx audio devices that exist in various laptops and peripherals. Most of last week was spent reading the code, and refactoring it to be a CX audio plugin in fwupd. There were a few things I could do to reduce the code size considerably:

  • Use the abstractions shared with all the other plugins, e.g. SREC file format processing, data chunking and low level USB HID
  • Drop support for hardware families which are no longer supported and not likely to receive updates
  • Remove the layers of abstractions and the macros-of-macros-of-macros so common with a codebase age measured in decades
  • Use helper objects in GLib and GObject rather than having to create everything from scratch

So, after all that we got down to a 1377 line fwupd plugin which is a 16x code reduction. It’s broadly comparable in functionality to the 22,000 line code drop but only works in fwupd as a plugin rather than as a standalone updater. To add support for new hardware to the plugin all we have to do is add an entry to the quirk file, which tells us which CX family the specific USB VID/PID is using. The rest is auto-detected.

I can’t tell you the OEM or the hardware all this work is being driven by, but eagle-eyed readers will work it out :) In some cases you might see an extra device appear in fwupdmgr get-devices if you’re running the soon-to-be-released fwupd 1.3.2 and hopefully we can get firmware updates which use this new device on the LVFS some time this year.

GNOME Firmware 3.34.0 Release

This morning I tagged the newest fwupd release, 1.3.1. There are a lot of new things in this release and a whole lot of polishing, so I encourage you to read the release notes if this kind of thing interests you.

Anyway, to the point of this post. With the new fwupd 1.3.1 you can now build just the libfwupd library, which makes it easy to build GNOME Firmware (old name: gnome-firmware-updater) in Flathub. I tagged the first official release 3.34.0 to celebrate the recent GNOME release, and to indicate that it’s ready for use by end users. I guess it’s important to note this is just a random app hacked together by 3 engineers and not something lovelingly designed by the official design team. All UX mistakes are my own :)

GNOME Firmware is designed to be a not-installed-by-default power-user tool to investigate, upgrade, downgrade and re-install firmware.
GNOME Software will continue to be used for updates as before. Vendor helpdesks can ask users to install GNOME Firmware rather than getting them to look at command line output.

We need to polish up GNOME Firmware going forwards, and add the last few features we need. If this interests you, please send email and I’ll explain what needs doing. We also need translations, although that can perhaps wait until GNOME Firmware moves to GNOME proper, rather than just being a repo in my personal GitLab. If anyone does want to translate it before then, please open merge requests, and be sure to file issues if any of the strings are difficult to translate or ambigious. Please also file issues (or even better merge requests!) if it doesn’t build or work for you.

If you just want to try out a new application, it takes 10 seconds to install it from Flathub.

Realizing that I’m not Super Human: Part 1

Most of the content on this blog is technical in nature, as is my twitter feed. I wanted to step to one side, and talk a bit about one of the little things I’ve learned about my body: I’m not super human any more.

I’m one of those people that have been really lucky with my general physical and mental health over the years. I used to play a lot of rugby and got the odd injury, but nothing a long hot bath couldn’t fix. Modulo catching the flu a few years ago I don’t really get ill very much.

About this time last year I began to get a small amount of back pain when sitting for a long time, or when walking around for over an hour or so. This was the first warning. Over the next few months this got worse to the point it was now an electrical tingling all down one leg whenever I did “too much” walking or playing with the kids. I self-diagnosed this as some kind of sciatica and didn’t pay too much attention to it. This was the second warning sign. After my back finally “went pop” a couple of times in one week leaving me unable to walk properly at all, I finally went to a private physiotherapist and asked for some advice. Luckily for me this was all covered as part of my Red Hat compensation package and I didn’t have to pay a thing, which I know really isn’t the case if you’re paying for healthcare yourself.

The Physio did quite a lot of tests and then announced that my posture was, put bluntly, total crap. There was no magic pill nor any special sports massage to make it better, but everything could be fixed with a little bit of hard work. I had to make some immediate changes: my comfy armchair was out, a standing desk was in. 10 hours sitting in a chair coding was bad, hourly breaks were enforced. I was given some exercises to do every day (which I did) and after about 6 weeks of visits I was discharged as the tingling had gone and the back pain was much less. The physio suggested I do a weekly Pilates class to further improve my posture and to keep everything where it should be.

This was waaaay outside my comfort zone, as I’d never done any kind of exercise or group class before. I went to a group class and immediately realized I was at least two orders of magnitude less capable than everyone else. I could barely touch my knees when they could all touch the floor. The instructor was really kind and showed me all the positions and things to do and not do, but I still felt a bit weird in a class of mostly middle aged women dragging them all down to my level. I asked the instructor if he did 1:1 classes and he said yes; Since then I’ve been doing a 1 hour Pilates class every other week and, against all odds, I’m actually quite enjoying it now. My posture is much better; when I run I feel less like I’m flopping about and now have a stable “core” of muscle holding me all together. I can throw my children around in the park, and not worry about discs in my back bulging to the point of rupture. My breathing and concentration has improved, and if anything I guess I’m slightly more productive with hourly breaks.

Talking to other men, it seems quite a few people also do Pilates, but for some reason are a bit embarrassed to admit it to other people. I suppose I was initially too, but not now. My wife does Yoga, and I guess to me Pilates feels like a more physical Yoga without all the spiritual stuff mixed in. I’m not quite a card-carrying evangelist, but I really would recommend you try Pilates if you sit at a desk all day hunched over an editor all day, like I used to. Doing 1:1 classes is expensive (about £80/month) but it is 100% worth it with the results I’ve had so far.

So, the conclusion: I’m not Super Human any more, but that’s okay. If you’ve read this far – shoulders back, chin up, and get back to coding. If you’re interested, want an awesome instructor and you live in West London, give Ash a call.

GNOME Firmware Updater

A few months ago, Dell asked if I’d like to co-mentor an intern over the summer. The task was to create a GTK “power user” application for managing firmware. The idea being that someone like Dell support could ask the user to run a little application and then read back firmware versions or downgrade to an older firmware version rather than getting them to use the command line. GNOME and KDE software centers deliberately show a “simple” view of firmware, only showing devices when updates are pending.

In June I was introduced to Andrew Schwenn, who was our intern for the summer. This blog isn’t about Andrew, but I will say he did amazingly well and was soon up to speed filing excellent pull requests even with a grumpy anally-retentive maintainer like me. Andrew has finished his internship now, but I wouldn’t be surprised if we work again with him in the future. Most of the work so far is from Andrew, so I can’t claim too much credit here.

GNOME Firmware Updater was designed in the style of a GNOME Control Center panel, and all the code is written in a way to make a port very simple indeed if that’s what we actually want. At the moment it’s a seporate project and binary, as we’re still prototyping the UI and working out what kind of UX we want from a power user tool. It’s mostly complete and a few weeks away from it’s first release. When it does get an official release, I’ll be sure to upload it to Flathub to make it easy for the world to install. If this sounds interesting to you the code is here. I don’t have a huge amount of time to dedicate to this power user tool, but please open pull requests or issues if there’s something you’d like to see fixed.