The future of package management in Fedora

I spent a couple of days last week in Brno talking about the future of package (and application) management in Fedora. Things we discussed:

  • We currently don’t cater for desktop users by showing details about the packaging layer (GPG keys, packages, etc) and not being able to search while we install/remove
  • We need a centralized application store to stay competitive with other distros and OSs
  • Our metadata and package mirroring policies hurt end users not using the command line
  • I presented (and demoed) gnome-software with its plugin architecture that would allow us to switch to using packages and blobs like glick and listaller
  • I made the case for application metadata, so we can get things like localised application details, screenshots and ratings using a few different methods

Some interesting points came up, and this is approximately 30 hours of talking condensed into a few bullet points:

  • YUM upstream will soon be considered deprecated, and we will move into a DNF/hawkey/librepo-based future. This includes PackageKit. I’m going to be building a hawkey based backend with help from the maintainer, and he is is aware of what PK does and any unusual tasks that are performance critical.
  • We should keep old versions of metadata on the server to stop metadata refresh explosions happening where yesterdays primary gets updated because a transaction only has todays filelists installed. This will significantly reduce the amount of bandwidth used by the metadata updates.
  • We should keep old versions of packages on the mirrors, to avoid the case where we depsolve fine, get 404 on the package download and then have to re-download MD, depsolve, etc. YUM apparently has issues with multiple versions of a package being present in the metadata, so we should probably only reference the latest packages in the MD (which also keeps the MD to sane size).
  • We should ship the per-arch solv files in the repo MD. This avoids SAT solutions like libsolv from spending 20 seconds+ per repo rebuilding .solv files from sqlite or xml metadata, and allows us to kill the dnf cron job
  • We should teach rpm to update it’s own SAT database, which we can do with an RPM plugin.
  • We want a software center, and fedora-tagger can provide the ratings/comments information. We might need an OCS server for screenshots, or can tie in screenshots with automated QA somehow.
  • We are going to teach koji about appstream data, so a simple extract script (to be written by me) can produce a .tar file of icons and a .xml file of translated descriptions at the end of each koji build
  • We are going to teach the compose tools to xmlmerge all the appstream .xml files and ship as appstream.xml
  • We are going to teach the compose tools to join all the tar files and ship as appstream-data.tar.gz
  • We are going to investigate the use of meta-desktop files to install a super-set of applications, e.g. KDE, or “Python developer” which allows for screenshots, ratings and all that stuff.

It was an interesting couple of days, and quite a few people will be ping’ed over the next few days to make some of what we discussed a reality. I’m convinced the changes we can make here will give us a slick and featureful application installer, something that can really be an asset for Fedora and RHEL. Comments, as always, welcome.

Auto-EDID Results [updated]

A couple of weeks ago I asked people to run a command which uploaded all their auto-EDID display profiles to me. This was a massive success with 1858 profiles being added to a large dataset. These were scanned by the cd-find-broken tool, and results plotted on my G+ page. As there’s been so much new data I’m updating the graphs:

edid-vendors

I’m actually using this data to make sure we show something sane in the client UIs. Some interesting vendors are not included, e.g.:

  • “System manufacturer”,3
  • “To Be Filled By O.E.M.”,4

edid-vendors-broken

This is a chart of vendors Doing It Wrong™ by including random data (or implausible data) as the display primaries.

edid-cmf

This shows what program created the Auto-EDID ICC profile. Unknown is probably a mixture of oyranos and also early versions of gnome-setting-daemon which didn’t set the extra metadata.

edid-vendors-noserial

Last graph I promise. This shows a chart of all the vendors who do not populate the serial number in the EDID blob. I’ll explain why this is bad.

When we construct the device ID for colord, we use the vendor{-model}{-serial} as part of the key.  This allows you to use different ICC profiles even if you’ve got two “identical” external panels attached. Without the serial number, “lenovo-foo” looks the same as “lenovo-foo” and colord treats them as if they were the same panel. This sucks if the panels were not bought at the same time and have identical backlight burn time. Ohh, and we can’t use the connection name (e.g. DVI-1) as it would suck if you had to reassign all your profiles if you moved the connector to DVI-2…

This isn’t always a disaster: Laptops. We only need the make and model to ensure this is unique on the system as you can’t typically have two internal panels installed. This explains the Lenovo, Samsung, Dell and Apple entries I think, so don’t get out the pitchforks just yet. Unfortunately there’s nothing in the ICC profile that says “this is a laptop” so we can’t be more selective and hence this graph isn’t super useful. But, even on laptops, vendors should really be doing something semi-sane with the serial number, even if it’s just the batch number.

A new 0.1.34 colord was released this week. Thanks again to everyone that uploaded profiles.

 

Auto-EDID Profiles Results

First, thanks for everyone that contributed ICC profiles. I’ve received over 800 uploads in a little under 24 hours, so I’m very grateful.

TLDR:

Total profiles scanned: 800
Profiles with invalid or unlikely primaries: 45
EDIDs are valid 94.4% of the time

This resulted in the following commit to colord:

commit 87be4ed4411ca90b00509a42e64db3aa7d6dba5c
Author: Richard Hughes <richard@hughsie.com>
Date:   Wed Apr 24 21:47:14 2013 +0100
    Do not automatically add EDID profiles with warnings to devices

I’ll explain a little about what these numbers and the commit means. The EDID reports three primaries, i.e. what real world XYZ color red, green and blue map to. It also tells us the whitepoint of the display. From basic color science we know that

  • If R=G=B, we should be displaying a black/gray/white color without significant amounts of red green and blue showing
  • The reported gamut of colors should not be bigger on real hardware than theoretical archive spaces like ProPhotoRGB.
  • If R=G=B=100%, then we should have a good approximation of D50 white
  • The temperature of the display should not be too cold (>3000K) or too warm (<10000K).

There are actually 11 checks colord now does on RGB profiles, similar to the checks done above. If any of the 11 checks fail, the automatically generated profile is not used. The user can still add it manually if they want and then of course it will be used for the monitor, but we don’t break things by default for 5.6% of users.

If anyone is interested, the results were generated by this program, and the raw results are available here. My personal take-home messages you can take from this file are:

  • Sometimes blue and green are the wrong way around (Samsung SyncMaster)
  • Vendors need to use something other than random binary data (AU Optronics)
  • If you don’t know what a whitepoint is, don’t try and guess (Sharp and Lenovo YT07)
  • Projectors generally don’t really know/care what values to use (OPTi PK301 and In Focus Systems)

There’ll be a new colord release with this feature released in the next couple of weeks.

Auto-EDID ICC Profiles

A favour, my geeky friends:

gnome-settings-daemon and colord-kde create an ICC profile based on the color information found in the EDID blob. Sometimes the EDID data returns junk, and so the profile is also junk. This can do weird things to color managed applications. I’m trying to find a heuristic for when to automatically suppress the profile creation for bad EDIDs, such as the red primary being where blue belongs and that kind of thing. To do this, I need data. If you could run this command, I’d be very grateful.

for f in $HOME/.local/share/icc/edid-* ; do
    curl -i -F upload=@$f http://www.hughski.com/profile-store.php
done

This uploads the auto-EDID profile to my webserver. There is no way I can trace this data back to any particular user, and no identifiable data is stored in the profile other than the MD5 of the EDID blob. I’ll be sharing the processed data when I’ve got enough uploads. If you think that your EDID profile is wrong then I’d really appreciate you also emailing me with the “Location:” output from CURL, although this is completely optional. Thanks!

Translated Color Profiles

In GNOME 3.10 we’ll have translated ICC profiles thanks to all the translators. This should make Alexandre Prokoudine happy indeed.

Screenshot from 2013-04-18 08:49:52

For applications, libcolord provides a CdIcc GObject if you don’t feel like dealing with wchar_t‘s and ‘mluc‘ objects yourself. Applications that deal with ICC profiles and want to get the localized versions of the description should probably look at this example code in C or in Python.

Comments, as always, welcome.

GNOME Software overall plan

I’ve been asked by a few people now to outline my plans for improving software installation in GNOME. I’ve started to prototype a new app called ‘GNOME Software’. It exists in gnome git and currently uses PackageKit to manage packages. It’s alpha quality, but basically matches the mockups done by the awesome guys in -design. It’s designed to be an application management application. GNOME PackageKit lives on for people that know what a package is and want a pointy-clicky GUI, so I’m not interested in showing low level details for power users.

Of course, packages are so 2012. It’s 2013, and people want to play with redistributable things like listaller and glick2 static blobs. People want to play with updating an OS image like ostree and that’s all awesome. Packages are pretty useful in some situations, but we don’t want to limit ourselves to being just another package installer. From a end-user point of view, packages are just an implementation detail.

So, I’ve been designing gnome-software to be pluggable. This means you can write an AppStream plugin to provide things like icons and screenshots for not-yet-installed software. You can write a plugin to ask ostree to update itself, and also a plugin to ask PackageKit to update a specific package. The idea is that we just run all the plugins in parallel when the user opens the dialog, and hide all the gutty details about the application update/install/removal itself. If installing packages falls out of favour we drop the PackageKit plugin, and instead write a plugin for ${distribution_system_of_the_year}.

I’ve done a quick technical outline below:

System Design

There are a few sticky issues I’ve not yet solved, like what happens if inksape is installed locally using Glick2 and also installed as a package. I suppose you’d get two entries in the results, and two things to update. Not sure. Ideas welcome.

There’s quite a bit of working code in git, but I didn’t want to write too much until I’d had some feedback from the community. Comments, suggestions and flames very welcome. Thanks.

 

ColorMunki Smile

I’ve just purchased a ColorMunki Smile so I can write a native colord driver for the i1 display class of hardware. I’ll base this on the Argyll CMS driver which is also GPLv2+ and this will mean we can get faster and more reliable readings by not spawning /usr/bin/spotread and trying to screen scrape the output. It also means we can support the newer LED backlights in the client UIs, which the smile supports.