A few months ago I gave a talk at OSFC.io titled Introducing the Linux Vendor Firmware Service.
If you have a few minutes it’s a really useful high-level view of the entire architecture, along with a few quick dives into some of the useful things the LVFS can do. Questions and comments welcome!
Yesterday I wrote a blog about what hardware vendors need to provide so I can write them a fwupd plugin. A few people contacted me telling me that I should make it more generic, as I shouldn’t be the central point of failure in this whole ecosystem. The sensible thing, of course, is growing the “community” instead, and building up a set of (paid) consultants that can help the OEMs and ODMs, only getting me involved to review pull requests or for general advice. This would certainly reduce my current feeling of working at 100% and trying to avoid burnout.
As a first step, I’ve created an official page that will list any consulting companies that I feel are suitable to recommend for help with fwupd and the LVFS. The hardware vendors would love to throw money at this stuff, so they don’t have to care about upstream project release schedules and dealing with a gumpy maintainer like me. I’ve pinged the usual awesome people like Igalia, and hopefully more companies will be added to this list during the next few days.
If you do want your open-source consultancy to be added, please email me a two paragraph corporate-friendly blurb I can include on that new page, also with a link I can use for the “more details” button. If you’re someone I’ve not worked with before, you should be in a position to explain the difference between a capsule update and a DFU update, and be able to tell me what a version format is. I don’t want to be listing companies that don’t understand what fwupd actually is :)
I’ve been told by several sources (but not by Google directly, heh) that from Christmas onwards the “Designed for ChromeBook” sticker requires hardware vendors to use fwupd rather than random non-free binaries. This does make a lot of sense for Google, as all the firmware flash tools I’ve seen the source for are often decades old, contain layer-on-layers of abstractions, have dubious input sanitisation and are quite horrible to use. Many are setuid, which doesn’t make me sleep well at night, and I suspect the security team at Google also. Most vendor binaries are built for the specific ODM hardware device, and all of them but one doesn’t use any kind of source control or formal review process.
The requirement from Google has caused mild panic among silicon suppliers and ODMs, as they’re having to actually interact with an open source upstream project and a slightly grumpy maintainer that wants to know lots of details about hardware that doesn’t implement one of the dozens of existing protocols that fwupd supports. These are companies that have never had to deal with working with “outside” people to develop software, and it probably comes as quite a shock to the system. To avoid repeating myself these are my basic rules when adding support for a device with a custom protocol in fwupd:
README.md file.0x12345678 meant to be presented as “12.34.5678” or “18.52.86.120” – the LVFS really cares if this is correct, and users want to see the “same” version numbers as on the OEM web-page.#defines or to add variations to the protocol for new versions of the hardware. I’m going to be grumpy if I just get sent a random email with demands about backporting all the VID/PIDs to Debian stable. I have zero control on when Debian backports anything, and very little influence on when Ubuntu does a SRU. I have a lot of influence on when various Fedora releases get a new fwupd, and when RHEL gets backports for new hardware support.Now, if all this makes me sound like a grumpy upstream maintainer then I apologize. I’m currently working with about half a dozen silicon suppliers who all failed some or all of the above bullets. I’m multiplexing myself with about a dozen companies right now, and supporting fwupd isn’t actually my entire job at Red Hat. I’m certainly not going to agree to “signing off a timetable” for each vendor as none of the vendors actually pay me to do anything…
Given interest in fwupd has exploded in the last year or so, I wanted to post something like this rather than have a 10-email back and forth about my expectations with each vendor. Some OEMs and even ODMs are now hiring developers with Linux experience, and I’m happy to work with them as fwupd becomes more important. I’ve already helped quite a few developers at random vendors get up to speed with fwupd and would be happy to help more. As the importance of fwupd and the LVFS grows more and more, vendors will need to hire developers who can build, extend and support their hardware. As fwupd grows, I’ll be asking vendors to do more of the work, as “get upstream to do it” doesn’t scale.
A month ago, GNOME was hit by a patent troll. We’re fighting, but need some money to fund the legal defense, and counterclaim. I just donated, and if you use or develop free software you should too.
A couple of weeks ago, Synaptics (who now own Conexant) sent me 22,000+ lines of LGPLv2+ licensed C++ that was capable of updating the firmware of all the CXxxxx audio devices that exist in various laptops and peripherals. Most of last week was spent reading the code, and refactoring it to be a CX audio plugin in fwupd. There were a few things I could do to reduce the code size considerably:
So, after all that we got down to a 1377 line fwupd plugin which is a 16x code reduction. It’s broadly comparable in functionality to the 22,000 line code drop but only works in fwupd as a plugin rather than as a standalone updater. To add support for new hardware to the plugin all we have to do is add an entry to the quirk file, which tells us which CX family the specific USB VID/PID is using. The rest is auto-detected.
I can’t tell you the OEM or the hardware all this work is being driven by, but eagle-eyed readers will work it out :) In some cases you might see an extra device appear in fwupdmgr get-devices if you’re running the soon-to-be-released fwupd 1.3.2 and hopefully we can get firmware updates which use this new device on the LVFS some time this year.
This morning I tagged the newest fwupd release, 1.3.1. There are a lot of new things in this release and a whole lot of polishing, so I encourage you to read the release notes if this kind of thing interests you.
Anyway, to the point of this post. With the new fwupd 1.3.1 you can now build just the libfwupd library, which makes it easy to build GNOME Firmware (old name: gnome-firmware-updater) in Flathub. I tagged the first official release 3.34.0 to celebrate the recent GNOME release, and to indicate that it’s ready for use by end users. I guess it’s important to note this is just a random app hacked together by 3 engineers and not something lovelingly designed by the official design team. All UX mistakes are my own :)
GNOME Firmware is designed to be a not-installed-by-default power-user tool to investigate, upgrade, downgrade and re-install firmware.
GNOME Software will continue to be used for updates as before. Vendor helpdesks can ask users to install GNOME Firmware rather than getting them to look at command line output.
We need to polish up GNOME Firmware going forwards, and add the last few features we need. If this interests you, please send email and I’ll explain what needs doing. We also need translations, although that can perhaps wait until GNOME Firmware moves to GNOME proper, rather than just being a repo in my personal GitLab. If anyone does want to translate it before then, please open merge requests, and be sure to file issues if any of the strings are difficult to translate or ambigious. Please also file issues (or even better merge requests!) if it doesn’t build or work for you.
If you just want to try out a new application, it takes 10 seconds to install it from Flathub.
Acer has now officialy joined the LVFS, promoting the Aspire A315 firmware to stable.
Acer has been testing the LVFS for some time and now all the legal and technical checks have been completed. Other models will follow soon!
Most of the content on this blog is technical in nature, as is my twitter feed. I wanted to step to one side, and talk a bit about one of the little things I’ve learned about my body: I’m not super human any more.
I’m one of those people that have been really lucky with my general physical and mental health over the years. I used to play a lot of rugby and got the odd injury, but nothing a long hot bath couldn’t fix. Modulo catching the flu a few years ago I don’t really get ill very much.
About this time last year I began to get a small amount of back pain when sitting for a long time, or when walking around for over an hour or so. This was the first warning. Over the next few months this got worse to the point it was now an electrical tingling all down one leg whenever I did “too much” walking or playing with the kids. I self-diagnosed this as some kind of sciatica and didn’t pay too much attention to it. This was the second warning sign. After my back finally “went pop” a couple of times in one week leaving me unable to walk properly at all, I finally went to a private physiotherapist and asked for some advice. Luckily for me this was all covered as part of my Red Hat compensation package and I didn’t have to pay a thing, which I know really isn’t the case if you’re paying for healthcare yourself.
The Physio did quite a lot of tests and then announced that my posture was, put bluntly, total crap. There was no magic pill nor any special sports massage to make it better, but everything could be fixed with a little bit of hard work. I had to make some immediate changes: my comfy armchair was out, a standing desk was in. 10 hours sitting in a chair coding was bad, hourly breaks were enforced. I was given some exercises to do every day (which I did) and after about 6 weeks of visits I was discharged as the tingling had gone and the back pain was much less. The physio suggested I do a weekly Pilates class to further improve my posture and to keep everything where it should be.
This was waaaay outside my comfort zone, as I’d never done any kind of exercise or group class before. I went to a group class and immediately realized I was at least two orders of magnitude less capable than everyone else. I could barely touch my knees when they could all touch the floor. The instructor was really kind and showed me all the positions and things to do and not do, but I still felt a bit weird in a class of mostly middle aged women dragging them all down to my level. I asked the instructor if he did 1:1 classes and he said yes; Since then I’ve been doing a 1 hour Pilates class every other week and, against all odds, I’m actually quite enjoying it now. My posture is much better; when I run I feel less like I’m flopping about and now have a stable “core” of muscle holding me all together. I can throw my children around in the park, and not worry about discs in my back bulging to the point of rupture. My breathing and concentration has improved, and if anything I guess I’m slightly more productive with hourly breaks.
Talking to other men, it seems quite a few people also do Pilates, but for some reason are a bit embarrassed to admit it to other people. I suppose I was initially too, but not now. My wife does Yoga, and I guess to me Pilates feels like a more physical Yoga without all the spiritual stuff mixed in. I’m not quite a card-carrying evangelist, but I really would recommend you try Pilates if you sit at a desk all day hunched over an editor all day, like I used to. Doing 1:1 classes is expensive (about £80/month) but it is 100% worth it with the results I’ve had so far.
So, the conclusion: I’m not Super Human any more, but that’s okay. If you’ve read this far – shoulders back, chin up, and get back to coding. If you’re interested, want an awesome instructor and you live in West London, give Ash a call.
A few months ago, Dell asked if I’d like to co-mentor an intern over the summer. The task was to create a GTK “power user” application for managing firmware. The idea being that someone like Dell support could ask the user to run a little application and then read back firmware versions or downgrade to an older firmware version rather than getting them to use the command line. GNOME and KDE software centers deliberately show a “simple” view of firmware, only showing devices when updates are pending.
In June I was introduced to Andrew Schwenn, who was our intern for the summer. This blog isn’t about Andrew, but I will say he did amazingly well and was soon up to speed filing excellent pull requests even with a grumpy anally-retentive maintainer like me. Andrew has finished his internship now, but I wouldn’t be surprised if we work again with him in the future. Most of the work so far is from Andrew, so I can’t claim too much credit here.
GNOME Firmware Updater was designed in the style of a GNOME Control Center panel, and all the code is written in a way to make a port very simple indeed if that’s what we actually want. At the moment it’s a seporate project and binary, as we’re still prototyping the UI and working out what kind of UX we want from a power user tool. It’s mostly complete and a few weeks away from it’s first release. When it does get an official release, I’ll be sure to upload it to Flathub to make it easy for the world to install. If this sounds interesting to you the code is here. I don’t have a huge amount of time to dedicate to this power user tool, but please open pull requests or issues if there’s something you’d like to see fixed.
CFU is a new specification from Microsoft designed to be the one true protocol for updating hardware. No vendor seems to be shipping hardware supporting CFU (yet?), although I’ve had two peripheral vendors ask my opinion which is why I’m posting here.
CFU has a bazaar pre-download phase before sending the firmware to the microcontroller so the uC can check if the firmware is required and compatible. CFU also requires devices to be able to transfer the entire new transfer mode in runtime mode. The pre-download “offer” allows the uC to check any sub-components attached (e.g. other devices attached to the SoC) and forces it to do dep resolution in case sub-components have to be updated in a specific order.
Pushing the dep resolution down to the uC means the uC has to do all the version comparisons and also know all the logic with regard to protocol incompatibilities. You could be in a position where the uC firmware needs to be updated so that it “knows” about the new protocol restrictions, which are needed to update the uC and the things attached in the right order in a subsequent update. If we always update the uC to the latest, the probably-factory-default running version doesn’t know about the new restrictions.
The other issue with this is that the peripheral is unaware of the other devices in the system, so for instance couldn’t only install a new firmware version for only new builds of Windows for example. Something that we support in fwupd is being able to restrict the peripheral device firmware to a specific SMBIOS CHID or a system firmware vendor, which lets vendors solve the “same hardware in different chassis, with custom firmware” problem. I don’t see how that could be possible using CFU unless I misunderstand the new .inf features. All the dependency resolution should be in the metadata layer (e.g. in the .inf file) rather than being pushed down to the hardware running the old firmware.
What is possibly the biggest failure I see is the doubling of flash storage required to do an runtime transfer, the extra power budget of being woken up to process the “offer” and enough bulk power to stay alive if “unplugged” during a A/B swap. Realistically it’s an extra few dollars for a ARM uC to act as a CFU “bridge” for legacy silicon and IP, which I can’t see as appealing to an ODM given they make other strange choices just to save a few cents on a BOM. I suppose the CFU “bridge” could also do firmware signing/encryption but then you still have a physical trace on the PCB with easy-to-read/write unsigned firmware. CFU could have defined a standardized way to encrypt and sign firmware, but they kinda handwave it away letting the vendors do what they think is best, and we all know how that plays out.
CFU downloads in the runtime mode, but from experience, most of the devices can transfer a few hundred Kb in less than ~200ms. Erasing flash is typically the slowest thing, typically less than 2s, writing next at ~1s both done in the bootloader phase. I’ve not seen a single device that can do a flash-addr-swap to be able to do the A/B solution they’ve optimized for, with the exception of enterprise UEFI firmware which CFU can’t update anyway.
By far the longest process in the whole update step is the USB re-enumeration (up to twice) which we have to allow 5s (!!!) for in fwupd due to slow hubs and other buggy hardware. So, CFU doubles the flash size requirement for millions of device to save ~5 seconds for a procedure which might be done once or twice in the devices lifetime. It’s also not the transfer that’s the limitation even over bluetooth as if the dep resolution is “higher up” you only need to send the firmware to the device when it needs an update, rather that every time you scan the device.
I’m similarly unimpressed with the no-user-interaction idea where firmware updates just happen in the background, as the user really needs to know when the device is going to disappear and re-appear for 5 seconds (even CFU has to re-enumerate…) — image it happening during a presentation or as the machine is about to have the lid shut to go into S3.
EDIT: See the new fwupd plugin.