Offline OS Updates – Looking forward to GNOME 3.6

All weekend I’ve been hacking on PackageKit and systemd to be able to have a GNOME 3.6 user experience that looks like the new update mockups.

So what’s the plan?

  • gnome-settings-daemon will automatically prepare the transaction using PackageKit downloading all packages (either all, or just security updates) and deps which in turns creates a /var/lib/PackageKit/prepared-update file when ready [works]
  • If /var/lib/PackageKit/prepared-update exists then gnome-shell will show a “Restart and install updates” option, that if clicked will call pkexec pk-trigger-offline-update which creates /system-update and the session is rebooted [needs gnome-shell patch]
  • On next boot, if /system-update exists, then the systemd generator starts system-update.target which in turn starts packagekit-offline-update.service, which in turn makes PackageKit run the prepared update transaction. On error, /system-update is removed, and on success both /system-update and /var/lib/PackageKit/prepared-update are removed. [works]
  • Plymouth will show a package icon (or something) with a widget that fills up as the transactions are processed (0 to 100%) [needs-work]
  • Plymouth will show a message after the updates are applied like “Rebooting after installing updates” [needs-work]
  • Show a message at next boot if the offline update succeeded or failed [working-on-right-now]

So why bother with all this?

  • Installing updates while the session is running causes havoc with some apps like firefox that have file resources that have not been locked (just try updating xulrunner when firefox or thunderbird is open…)
  • Installing library updates when apps are running against the old copies means the processed need to be restarted (gnome-session, sshd, etc) before the changes are in effect (for all users logged into the machine)
  • Installing core OS updates and doing OS upgrades in the running session works for most people most of the time, and then when it fails it destroys your system completely with no way to recover
  • Using a minimal pre-boot environment we can snapshot the system before we update the OS and afterwards (requires btrfs or something else)
  • Using a fresh pre-boot environment means we can easily check OS sanity before we start updating core bits of the OS, without lots of additional processes running.

Of course, we’ll still support updating applications in the session for GNOME 3.6 (as long as they are not running) just not the core OS bits. Comments, as always, welcome.

Published by

hughsie

Richard has over 10 years of experience developing open source software. He is the maintainer of GNOME Software, PackageKit, GNOME Packagekit, GNOME Power Manager, GNOME Color Manager, colord, and UPower and also contributes to many other projects and opensource standards. Richard has three main areas of interest on the free desktop, color management, package management, and power management. Richard graduated a few years ago from the University of Surrey with a Masters in Electronics Engineering. He now works for Red Hat in the desktop group, and also manages a company selling open source calibration equipment. Richard's outside interests include taking photos and eating good food.

73 thoughts on “Offline OS Updates – Looking forward to GNOME 3.6”

    1. I hate this :)

      Once a few packages start doing this, it can easily creep to other packages because it is easier on developers. At that point, we’ll end up having to reboot systems much more often, causing system downtime that is often not going to be acceptable.

      Once the price of applying a security update becomes system downtime, many systems will end up behind on security updates. This update model cannot become mandatory.

      1. It’s never going to become mandatory. Admins will still be able to do “yum update foo” and then update services when is convenient. We’re just not going to expect desktop users to do this.

  1. Reliable snapshosts are a nice dream, but I can’t see how that works except for the trivial cases.

    The trivial case is “After reboot, /bin/sh crashes” – in that case it is reasonably safe to revert to a previous snapshot.

    But once you let a user login, you don’t know what to snapshot/revert. /etc/shadow is modified by post-install scripts, so it needs to be reverted, but it also contains users passwords, so it needs to be kept. The same thing happens for just about every configuration file, and many files in /var – but not all, e.g. the rpmdb needs to be consistent. Add format changes on package update, and…

    I’d be interesting in seeing even a written description of what “snapshot/restore” is exactly supposed to snapshot/restore (without resorting to hand-waving like “the system”); I can’t see this having an easy answer.

    Then comes the implementation part, separating the files that need to be reverted from files that need to be kept, which is a “boiling the ocean” kind of problem.

    1. You just don’t get it. Snapshots are blocklevel, look at Solaris “boot environments” on top of ZFS. They got i right and Linux plays catchup with a series of half-baked attempts (as usual). If we are lucky we get a decent implementation in a few years but don’t hold your breath.

  2. hmm… I don’t know. I’m not really looking forward to these changes. My uptimes are usually very long (last time, it was like 110 days) on my personal laptop. And I really don’t want to be forced to reboot my machine every time some updates arrive. It’s so Windows-like.
    I’ve never broken anything during updates. What concerns me more is that nothing stops the system from rebooting while updates are being installed on the background. This is something that can really break your system and no one has fixed it even though it’s a known bug for ages.

    1. Long uptimes aren’t cool anymore. They mean you are applying kernel fixes and updates. The cool thing is to have a 2+ node highly available virtual machine cluster, that you can live migrate your nodes back and forth between. This way, your vm’s stay up while the physical nodes get to take turn doing updates. As for rebooting the vm’s themselves, they should be HA somehow, (eg: two dns servers) so that you can take turns rebooting them too.

      Happy hacking!
      PS: I don’t like rebooting my laptop either, but i probably wouldn’t mind if gnome-session existed and brought all my terminals back the way they were!

  3. I like the idea of snapshots a lot, although I think the lack of this ability with most file systems decreases the value of this ‘reboot twice’ approach. I think this makes a lot of sense for mobile devices, but if ‘core bits’ includes libraries, I think many developers and users will be annoyed with it, even if they need a rollback of some sort. Of course, being ‘failsafe’ is more important than being convenient, sometimes.

    However, if this is only necessary on kernel updates and core GNOME infrastructure (like every six months), I don’t think anyone would mind too much.

    I know my personal case isn’t common, but I use Arch Linux, a rolling release where one of the main benefits is getting the most current stable software as it becomes available. If every update to libraries like 7zip, virtualbox modules, or xulrunner requires a reboot, I’ll probably be pretty annoyed. Then again, I can always update with pacman instead of PackageKit if I find it burdensome- perhaps this isn’t typical enough to be concerned about.

  4. Would be cool if instead of just rebooting twice, it did:
    init 1
    install updates
    if upgrades contains kernal or init (systemd):
    reboot
    else:
    init 5

      1. You know, sir, there is this thing called kexec, capable of upgrading kernel *without* reboot. But of course, that is no good for Gnome OS and packages which are much more deeply integrated into system compared to Linux kernel – examples being FF and Chrome. >-)

        On that note, another example of fail would be this DNF thing – because reinventing the wheels instead of using zypp just rocks, not to mention the wonderful python dependency.

  5. Instead of/Additionally to a “Restart and install updates” option I would like to automatically install updates in the background to a new btrfs snapshot.

    After the normal installation process is finished, all updated process should be “restarted”, which means that they need to be told to store the session data somewhere, kill themselves and start their new version from the new btrfs snapshot with the the saved session data.

    If a process doesn’t support the foresaid “invisible restart” feature, the process will still see the old btrfs snapshoot as long as it runs. When you close the process and start a new one, the new process will use the data from the new btrfs snapshoot.

    This way no user will be bothered by reboot requests and no process will complain about unwanted data changes.

  6. Have you considered the situation when I need to boot the computer super-fast to e.g. check when my train leaves and I can’t wait 10 minutes for a system update? Will I be able to skip the update process, or will it be the same as the “god dammit, bloody Windows updates” experience?

    1. We probably need to have some kind of “Press Escape to cancel” type thing too. If you file it, I’ll look at it.

        1. Same problem, what if I need to shutdown fast my laptop before placing it in my bag :)

          1. I imagine there will be two options: shutdown and “install updates and reboot/shutdown”, kind of like OS X does it.

        2. Then you may have problems when you shutdown because your battery is low (you really don’t want to run out of battery during say glibc update) or you are asked to “turn off all electronic devices”. Also if you want to check when your train is leaving you would mind upgrading on shutdown.

        3. No, it’s the same problem. Sometimes you need to boot fast, sometimes you need to shut down fast.

        4. Without a cancel option it will have the same problem. You want to shutdown but have to wait for updates to complete.

        5. Same problem, if you want to shut down your computer or reboot into a different OS before you catch the train.

  7. Finally! I always thought it was crazy to update applications while they where running.

  8. This looks cool. What all is included in “Core OS bits”. Will updating Gnome shell require a reboot when i could just as well log out and log back in.

  9. I though that systemd with its use of sockets was supposed to restart services (daemons) without any disturb for the user. I also always thought that UNIX is able to keep in memory an “old” file, including a library, so it can be replaced on disk immediately.

    To be honest, I like the cleanliness of the install process but I really don’t want to be forced to reboot my computer. And most of the time, I don’t even logout-login when PackageKit asks me to do so because I know from the list of updated packages that it is useless: I just have to close and re-run a program (if I really want to see the new version — and if it is Firefox for example ;-)).

    I would prefer to see work focused on avoiding any reboot or logout but not to make these processes “nice”…

  10. This is back to front.
    Updates which need a restart should be installed on close-down not on the next start-up.

    If the user is going to continue using the computer after the update, then those updates can be made before the system closes down or when it starts up. It makes no difference to the user since his/her time is wasted anyway.

    If the user deferred the updates until he/she was finished (a common scenario), then the updates should be installed as part of the close down procedure. It just takes a little longer for the computer to close down after the user has moved onto something else anyway. When the machine is next switched on, it will boot immediately rather than leaving the user waiting to start whilst the updates are carried out.

    For most people, who can defer system updates until the end of the day, this provides a much better experience.

    p.s. I remember having to switch on my Windows computer half an hour before I wanted to use it every morning, because of the update & reboot procedure that invariably stopped me from getting started on my work. Its one of the many reasons that I stopped using Windows. Please don´t port that behaiour to Linux.

  11. This offline-updates feature, if made mandatory, will kill both Debian and Gentoo. Are you sure you want to do this without coordinating with their maintainers? The conflicting features are interactive post-install scripts and the general concept of installing packages from source one-by-one.

    In Debian, package installation (or, more precisely, postinst scripts) may be interactive. If this is a remote server, how do I connect to the post-install script to provide the required input? This applies, e.g., to all applications that use a database via the dbconfig-common package and have SQL files in a standard location – if the SQL fails they ask what to do. The solution would be to extract and run all the config templates before the reboot, and ban using interactive features of debconf in other scripts. This does require dropping some error-handling features from dbconfig-common.

    In Gentoo, there is no PackageKit. However, if it existed, your proposal would amount to compiling all packages between the first and the second rebot – i.e. ~10 hours of downtime if both Chromium, Wine, Inkscape and OpenOffice happen to be updated at the same time. In other words, Gentoo users rely on their system to be usable while compiling and installing packages one-by-one. The solution would be to use btrfs (once it becomes usable) and compile/install all updates to a writeable snapshot, as already suggested in another comment.

    1. I am at a loss as to how a change to Fedora would kill completely separate distributions? Or how Fedora could make offline-updates mandatory for other distributions?

  12. Must be a gnome 3 specific problem… After doing live updates on multiple Linux systems for over 15 years, the last 8 including Ubuntu with the default gnome install I’ve never experienced anything worse than a browser malfunction when I’ve updated the running browser.

    Because of that I will sometimes restart some complex apps, but not terminal windows or the system. I’ve gone over a year doing updates multiple times per week without rebooting. And that including replacing the kernel on the disk but of course still running the in-memory version.

    I’ve found updating a Linux system to be far and away the best experience compared to Windows or OSX, and now you want to change it?

    1. I can’t but agree:
      what’s wrong with the current update procedure?
      Maybe it could be something more fancy and/or under the hood BUT, in order to rock even more, it got to have some features.

      1 – the ability to decide whether or not to start an upgrade
      2 – the ability to stop an already started upgrade
      3 – the ability to choose if the upgrade must be started/completed on the next boot or on the next shutdown
      4 – a snapshot of the system just before the upgrade would still be great

  13. Hopefully I’m missing something but is it so that for a random user to get Firefox updated two reboots will be needed? Please correct me that I’ve mistaken as that sounds worse than WinXP/IE updates…

    1. The first update reboot is typically only a couple of seconds in duration. We’re not starting up much stuff at all, and rebooting automatically (also, not shutting much stuff down).

  14. Congratulation!

    Now you force people to reboot just to install an upgrade, a big step on taking the worst of Windows.

    Instead of trying to write correct upgrade procedures and applications capable to stand an upgrade without failling you solve things imposing unneeded reboots.

  15. I’ve always HATED Windows which forces me to wait long either when rebooting/shutting down or when booting; or even both, before I can use my system. And I’d hate PK if it “forces” me to reboot and/or wait for updates to be installed on boot before it lets me to work with my system. It would be nice if PK “suggests” me to reboot, and “suggests” me to do an “offline install”, but forcing me to do so is not interesting for me.
    Also, it should be carefully decided which packages really benefit from “offline update”s. Certainly, kernel doesn’t. There is absolutely NO HARM in installing an updated kernel while you are working with the system. And it’ll save you some time (you don’t need to wait for the kernel to be installed before you can use your system).
    Please please please don’t make Linux like Windows. :(

  16. what a stupid idea. You should have rather spent your time figuring out the reason why updating a running app fails in the 1% of the cases instead of just wrecking 100% of the use cases.
    BTW firefox shows a restart button when it got updated giving the user the choice to do micro-reboots…

    1. On what operating system, on Windows?

      By the way I’ve always had problems with Firefox on Fedora if I upgraded it while it was running. The menus and the mouse (if I remember correctly) would would stop working.

      1. No, on Ubuntu Firefox always shows a restart message after an update was installed.

    1. You’re joking, right? Nix just works around the linked library issue with a set of hacks. You can’t deliver a monolithic OS appliance using scraped sha sums of runtime deps from binaries…

  17. All your ideas should be implemented as optional feature, because if you screw up distros like debian or gentoo, you will loose the last experts. There will be the regular users using gnomedows3.99 and the skilled users avoiding gnome wherever possible. Gnome is becoming worse with every version, please stop that.

  18. Given that larger file systems will require a more reliable filesystem, we can see that btrfs or zfs will be adopted. These filesystems support snapshots.

    If running on a current fs, the normal “unlink but still in use” already supports superior updating. The one use-case that a reboot on update policy has is where the in-use component that has been unlinked but is still in use doesn’t work with a on-disk component that will be used.

    This issue vanishes with a snapshot approach, making a forced reboot moot. However, identifying applications that can be updated “easily” vs those that are “difficult” would be useful. The idea of segregating via “does it install a desktop file” doesn’t work, specifically Firefox (the application of interest) does install one of these, so I don’t see your solution as working.

    However Firefox is capable of updating itself, and this can be exploited by providing a registration API for this class of application. If this registration API is used, and the application is in use, an attempted update could offer an automatic shutdown/restart of the affected application (firefox/sshd/etc.)

    But, again, this will soon be a moot point. I don’t think dev effort is well-expended in this direction, until btrfs is finalized.

  19. I have been using various Fedora systems daily since 2004 and have never had problems with updates, except the occasional bug in a newly updated package that is bad enough to require reverting to the previous version.

    If a new package is bad, will this new system give you a way to revert it and wait for the next release, or will it continue to add the bad package to the off-line installation list?

    I thought that Fedora was a test-bed for Red Hat Enterprise Linux. Daily reboots of a personal system running Fedora are one thing, but daily reboots of a server running RHEL are another matter. The goal should be reducing reboots through tools like ksplice. Requiring frequent reboots to keep systems updated is a big step backwards.

  20. Oh, nice thing. Does itmean now that:

    1. Updates that require a gnome shell logout now will likely require a reboot now (in general, more updates will require a reboot!)
    2. People have to interrupt the workflow to have a working computer after an update
    3. We are aiming at Windows XP compatibility with a somehow nicer look and feel

    It would be better to focus on solving real problems, because Gnome 3 has many… and also understand the big difference between Desktop computers and Mobile computers.

    1. We ship waaay too many updates at the moment. Something like a patch Tuesday for regular updates with out-of-band security updates (or the ability to install bugfiz packages manually out-of-schedule) is probably what makes sense on a desktop OS. The continued stream of nonsense updates isn’t sustainable at all.

  21. This is like windows updates, “reboot to install updates, and please don’t touch anything”. I’m amazed that gnome-shell is given the same importance as sshd and even more, that I’m forced to reboot to have the newer kernel working. What if I want to upgrade the kernel but don’t use it until I want/need to? This is definitively taking a direction in the wrong way. This has forced me to move to another linux distro. It’s a very big mistake to mix gnome with linux, gnome should not take precedence over the kernel.

  22. Congrats, this now officially sucks twice much as Windows, which at least is smart enough to do what duncan suggests and needs just one reboot, i.e. “Updates which need a restart should be installed on close-down not on the next start-up.” So, because of a bunch of semi-broken apps, you now require people to reboot their OS twice. Well done, like with other Lennartware. Epic fail.

  23. this is so cool, I will be restarting my linux desktop so often. can’t wait to do it again

  24. > Installing updates while the session is running causes havoc with some apps like firefox that have file resources that have not been locked (just try updating xulrunner when firefox or thunderbird is open…)

    Yes, that’s valid use case. Now somebody has to explain why to restart machine because of one application? Expecialy for Firefox simple plugin which will tell user: “hey FF is going to be updated and it has to be closed for a while” seems to be much better choice.

    > Installing library updates when apps are running against the old copies means the processed need to be restarted (gnome-session, sshd, etc) before the changes are in effect (for all users logged into the machine)

    Yes this requires reboot. I really don’t understand that this can be argue. Extension of RAM requires poweroff. Is this also support this feature?

    > Installing core OS updates and doing OS upgrades in the running session works for most people most of the time, and then when it fails it destroys your system completely with no way to recover

    This seems to be really same issue for proposed feature. Actually it can be even worse, something really near to disaster can happen if that feature will provide just limited environment by any means.

    > Using a minimal pre-boot environment we can snapshot the system before we update the OS and afterwards (requires btrfs or something else)

    Hmm and Picard’s enterprise does it in same way, right? Despite to fact I like sci-fi I would like to see real use cases rather.

    > Using a fresh pre-boot environment means we can easily check OS sanity before we start updating core bits of the OS, without lots of additional processes running.

    What to check? How to check? What’s the problem which is supposed to be fixed here?

    Well last question sums it at all. Where’s the bug which should be fixed by this bug and why it requires just this fix? I didn’t get it from this blogpost unfortunately.

    1. >Where’s the bug which should be fixed by this bug

      Just trawl bugzilla for the few hundred cases of corrupted rpmdbs. Usually live updates go fine, but when they go wrong, they blow away half of your system. That’s unacceptable for a desktop OS.

  25. This might be good, but in the whole article I cannot find good explanation why. If you want to introduce such level changes to OS, you really need to explain (and I mean in good technical manner):

    – why this is needed (what is currently broken)
    – will this be optional framework
    – will all or only some updates go through this way

    Otherwise the BFU will cause a havoc and completely reject the idea. Already now the forums are full of negative comments about fedora pulling some windows shit into the linux world.

  26. Still why reboot for updates?
    Cant there be way to take linux to init3, apply updates & take back to init5

    I hate reboot, why be like MS / MAC?

    1. Sure, you can do that if you want. In systemd it becomes even easier, but the general consensus was that we really needed to do this from a known-good boot. I suggest you take this up on the systemd mailing list if you’ve got a better idea on how to deal with the extra reboot. Thanks.

Comments are closed.