The future of package management in Fedora

I spent a couple of days last week in Brno talking about the future of package (and application) management in Fedora. Things we discussed:

  • We currently don’t cater for desktop users by showing details about the packaging layer (GPG keys, packages, etc) and not being able to search while we install/remove
  • We need a centralized application store to stay competitive with other distros and OSs
  • Our metadata and package mirroring policies hurt end users not using the command line
  • I presented (and demoed) gnome-software with its plugin architecture that would allow us to switch to using packages and blobs like glick and listaller
  • I made the case for application metadata, so we can get things like localised application details, screenshots and ratings using a few different methods

Some interesting points came up, and this is approximately 30 hours of talking condensed into a few bullet points:

  • YUM upstream will soon be considered deprecated, and we will move into a DNF/hawkey/librepo-based future. This includes PackageKit. I’m going to be building a hawkey based backend with help from the maintainer, and he is is aware of what PK does and any unusual tasks that are performance critical.
  • We should keep old versions of metadata on the server to stop metadata refresh explosions happening where yesterdays primary gets updated because a transaction only has todays filelists installed. This will significantly reduce the amount of bandwidth used by the metadata updates.
  • We should keep old versions of packages on the mirrors, to avoid the case where we depsolve fine, get 404 on the package download and then have to re-download MD, depsolve, etc. YUM apparently has issues with multiple versions of a package being present in the metadata, so we should probably only reference the latest packages in the MD (which also keeps the MD to sane size).
  • We should ship the per-arch solv files in the repo MD. This avoids SAT solutions like libsolv from spending 20 seconds+ per repo rebuilding .solv files from sqlite or xml metadata, and allows us to kill the dnf cron job
  • We should teach rpm to update it’s own SAT database, which we can do with an RPM plugin.
  • We want a software center, and fedora-tagger can provide the ratings/comments information. We might need an OCS server for screenshots, or can tie in screenshots with automated QA somehow.
  • We are going to teach koji about appstream data, so a simple extract script (to be written by me) can produce a .tar file of icons and a .xml file of translated descriptions at the end of each koji build
  • We are going to teach the compose tools to xmlmerge all the appstream .xml files and ship as appstream.xml
  • We are going to teach the compose tools to join all the tar files and ship as appstream-data.tar.gz
  • We are going to investigate the use of meta-desktop files to install a super-set of applications, e.g. KDE, or “Python developer” which allows for screenshots, ratings and all that stuff.

It was an interesting couple of days, and quite a few people will be ping’ed over the next few days to make some of what we discussed a reality. I’m convinced the changes we can make here will give us a slick and featureful application installer, something that can really be an asset for Fedora and RHEL. Comments, as always, welcome.

Published by

hughsie

Richard has over 10 years of experience developing open source software. He is the maintainer of GNOME Software, PackageKit, GNOME Packagekit, GNOME Power Manager, GNOME Color Manager, colord, and UPower and also contributes to many other projects and opensource standards. Richard has three main areas of interest on the free desktop, color management, package management, and power management. Richard graduated a few years ago from the University of Surrey with a Masters in Electronics Engineering. He now works for Red Hat in the desktop group, and also manages a company selling open source calibration equipment. Richard's outside interests include taking photos and eating good food.

40 thoughts on “The future of package management in Fedora”

    1. Yup, big stuff there! don’t forget to have a bugzilla product =)

  2. What would be cool, would be also to ease the policy in order to accept some free software packages implementing patented (often only relevant for the US) technologies, this is hurting end users way more than the way you access the packages IMHO.

    1. I think my hands are tied on the patent issue as Red Hat is a US company. Canonical can only get away with what they do as they are based in the Isle of Man.

    1. Basically, soon, Fedora will soon be shipping Appstream XML as metadata, and the software center will be consuming it.

      1. So does that mean that Fedora will be using AppStream? I don’t get it. How does AppStream (the supposed unified package installer) fit into what you guys are working on, or are they the same project?

        …and are other distros (namely .deb) users adopting this at the same time?

        AppStream seemed so promising at one time, but has since dropped off the face of the media. You hear nothing about it anymore.

        1. This is Appstream. We’re going to be generating it on the server and consuming it in the app installer. Appstream is really just an XML schema, so it’s hard to get excited over it. It is, however, slowly gaining traction.

          1. Good news! Thank you brother, may your days be long and prosperous!

            I hope it is only a short time before Mint/Debian and Arch? go this way too… then we’ll be unified finally. Ubuntu is doing its own thing lately, so we won’t include them in our hopes. Tee hee :)

          2. Yup, AppStream is the XML scheme as well as Ratings&Review, Screenshots etc. For cross-distro software installing, there is Listaller, which integrates with AppStream and PackageKit. (But Listaller is no “official” Freedesktop effort, for various reasons)
            At the time, all projects could need a lot more contributors from different distributions (this is what is lacking at time…).

  4. Hughsie,

    what does ‘YUM upstream will soon be considered deprecated, and we will move into a DNF/hawkey/librepo-based future. This includes PackageKit. I’m going to be building a hawkey based backend with help from the maintainer, and he is is aware of what PK does and any unusual tasks that are performance critical.’

    mean?

    Could you elaborate more?
    As Fedora is known as testbed for new RHEL technologies, how will that impact RHEL (7?, 8?) Admins?

    Will this change being incorporated in a repo admin system like Pulp?

    Would like to know about those plans.

    Thanks,

    \sh

    1. Hey, for RHEL 7 we’re still going to be using yum and all the usual stuff. For RHEL 8 I’m not sure. I think the idea is to slowly get things off yum and onto hawkey/dnf before Fedora 22.

      1. IIUC, the transition process is that dnf will be renamed into the new yum. Is that correct?

        1. I don’t think so, although Ales might know more. DNF isn’t going to present the yum api, so it would be odd calling it yum indeed.

          1. Since the command line options and configuration format is the same, it would make more sense for dnf to be renamed to yum IMO. Only developers care about the API change. Why bother users with it.

          2. +1 to Rahul’s comment below. It would save a lot of user confusion and “I hate change” dramatics. In a way, it *is* yum anyway.

  5. This is great news! Thank you for the update :) If you need any extra stuff for the AppStream part, I’ll help. Also, an AppStream-based software-center, which is not as slow as the USC, is currently being created (written in Vala).

  6. Noo! Yum don’t go! (Slow motion) :P

    Well. I undestand if it calls for better performance in Fedora, so thumbs up! Go for it! I’m already familiar with YUM though :(

    2. dnf is just “faster yum” for most people. Users will barely notice the difference, except for the nice performance improvements.

  7. Can you please solve the intel sandy bridge power regression problem on fedora (from the past 2 years till now) and then change the rpm?!

  8. Hi,
    Thanks!. I look forward these changes!!. =)

    A off topic.. In that state is zif?. I hoped that you try to use this, but you talk to replace yum with dnf..

    1. Well, zif is still available for use, but I think a SAT based solver is the way forward. Zif, as always is a ghost project :)

  9. Did you happen to see Ubuntu’s discussion about “Click Packages?” (https://launchpad.net/click-package, https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037074.html). If you haven’t, I suggest you take a look, because there might be some overlapping ideas. That project is in very very early stages, but I’ll be very excited if the major distros are aware of each other’s efforts on this front so we can maybe have some strong collaboration around application packaging in the future ;)

    1. I’ve not been impressed with Ubuntu recently. They’ve basically re-invented listaller in the click packages proposal, and after the Mir debarcle I’m wondering what’s next.

      1. This is the first time I’d heard of listaller (I had heard of Autopackage, however). I was curious as to why Ubuntu would come up with Click package given listaller already exists after reading your comment. There’s quite a lot of discussion around that in the mailing list thread that Dylan shared, in particular https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037079.html, which seems to suggest that Listaller is tied pretty closely to RPM. That might have done it.

        1. Well, Listaller is not tied to RPM or DEB, so this is not the reason. It’s package data is in fact very close to the one used for writing Debian packages.
          But Listaller is tied to PackageKit, and Ubuntu does not use PackageKit, but an own compatibility layer, which does not work with Listaller. (althought it could be made working, of course)

  10. Yep, that’s how it should be done. On the desktop. At least.
    DPKG vs RPM-related saga must come to the end for the desktop users, they must see same UI on all distros.
    Canonical made very bad thing by putting it’s own Ubuntu-only app manager. Working underground without OSS world is very wrong. There is a common middleware – packaging layer. Upper layer software must be able to work with everything. It’s not a kernel feature, which can run on Linux and won’t run on *BSD (sorry for that word folks).

    1. That attitude in Ubuntu is not about to change any time soon. Mark Shuttleworth is too much of a Crony Capitalist.

  11. p.s. hughsie, “They’ve basically re-invented listaller in the click packages proposal”.
    Nope, it’s not their invention. Opensuse had same thing several years ago too, and it has click-installation support now, which is actively used with repos, not as separate library/app mess like in Mac OS X. And don’t forget about CNR too, there were several universal, distro-independent attempts too.

  12. Ubuntu has a perfect app store which though cannot be compared to an Android one but its light years ahead of other distros. The mir part as long as they release the code in GPL and have good performance, what’s the big deal. I hope fedora solves all these installer problems, with F19 beta there are still basic problems with the installer and i hope fedora also shows some love to the desktop users which i feel Ubuntu and Unity are doing better than gnome

  13. It is amazing to see line here: “…hurt end users not using the command line…”. This shows your careness for your users.

    But maybe you should add a new line “..hurt end users don’t have internet…”. And I think you should add something like (or better than) Mintbackup on Linux Mint. Your package management system should able to backup what software you have installed. And user should able to choose what software, **not whole**. After backup, the software in one folder can be copied into another Fedora, and install it there. So, no internet again for another Fedora to install.

    This is the biggest thing missing (distribute the software installer/backup choosen installed software) in Linux I feel for years. Just my opinion.

  14. Awaiting changes and happy for the focus on features that would allow fedora to shine.

  15. This is great news! Looking forward to it. I use zif sometimes so I was curious to know, how would zif be related to this initiative ?

  16. why leaving yum instead of tweak that software ?
    I’ve use Fedora for 4 years and after this change I’ll leave Fedora for sure. And also about sofware center, did Fedora want to challenging Ubuntu?

    I’m just regular user who love using Fedora, but after too much changes in Fedora lately, I’ll gone and move to rolling release distro.

    thanks for take this out :D so I can leave earlier

    1. “I’ve use Fedora for 4 years and after this change I’ll leave Fedora for sure.”

      Cool amateur dramatics, bro. dnf has almost exactly the same set of commands as yum. It just has different internals to make it faster, lighter on bandwidth, more correct etc. Just try it for 10 seconds and you’ll see that. Are you really prepared to change distro just because someone tried to make yum faster and better? I think it’s more a case that you’re too lazy to learn something with *very* little extra learning curve and too willing too whine like a baby.

  17. Sounds very interesting. Any new install tools must support the following use cases:
    – simple end users with GUI, screen shots, etc. (think Android market)
    – servers using scripting or command line (update of YUM)
    – disconnected systems (systems not on the Internet, mirrored updates/packages)
    – near-seamless support of 3rd party and local repos
    — corporate or project-specific
    — isolated systems with local repos
    — cached repos
    — repos supporting patented software

  18. Sounds like a great roadmap. The only thing that worries me is the insistence on everything being XML again. The amount of pointless serialization bloat and redundancy in the XML metadata is absolutely insane.

    The excuse that “XML compresses well, so it’s not a problem” is completely and utterly bunk too. It still has to be extracted, read, parsed and stored in memory. XML is a universal solution to a non-existent problem and is an absolutely horrible waste of resources, especially on low end hardware that doesn’t have the luxury of being wasteful.

Comments are closed.