Introducing display calibration using colord-session

This year I never got around to using my annual holiday. To use up some of the time that I cannot transfer over to next year I took the last 4 days off work to put up Christmas lights, buy a Christmas tree and create a new bit of shared infrastructure for calibrating displays.

I’ll describe how we calibrated displays in GNOME 3.4 and GNOME 3.6:

  • User opens the color control panel and clicks on the device, then “Calibrate”
  • We spawn the gcm-calibrate executable with the colord device-id as an argument
  • The user goes through a GTK wizard, and we calibrate the display to a target whitepoint and then characterize the display.

To do the last bullet point involved spawning dispcal and dispwin in a VTE widget and screen-scraping the output from ArgyllCMS. This would break every release and was really fragile. Also, dispwin and dispcal would show it’s own GL window with the sample window, so we’d have bugs where the window was on the wrong output (as argyllcms sometimes didn’t match the xorg display numbering) and also where the sample window was covering up the instructions in the GtkWizard. And if either of the tools crashed or the sensor was removed before finishing, then undefined things happened, ranging from a crash to a disastrous hang where the user couldn’t even click the Quit button. And to top it all off, there was no sensible progress completion support, so the user basically didn’t know if the calibration was taking a long time, had crashed or was waiting for input in the (hidden) VTE widget.

Basically, it was a hack, and just not good enough.

KDE did something similar. So much so, it used the same gcm-calibrate binary. This meant you switched from the native KCM in KDE and were popped into a GtkWizard with the same separate GL window issues. This meant you had to install quite a few of the GNOME libraries just to be able to calibrate your display in KDE.

So, not good enough, squared.

So I did three things in the last week or so:

  • Added a “spotread” sensor type in colord. This sandboxes the argyllcms spotread process and abstracts out any undesirable behavior. You can read more about it here.
  • Added a CdSampleWidget object to libcolord-gtk for simplicity (e.g. cursor hidden at the right time)
  •  Created a colord-session D-Bus activated process (that only gets started when a display gets calibrated and quits as soon as the caller quits or calibration finishes). This does all the heavy lifting in the session.

The latter has signals like ::UpdateSample() to update the color sample on the screen and ::UpdateGamma() to update the 2D LUTs which allows a small D-Bus wrapper to show any kind of GUI it likes. This lets us implement the new GNOME design mockups designed by Allan which look a lot less scary than the old UI. We can also pass a device-id and sensor-id to the new Start() method which means the calibration window gets shown on the right screen 100% of the time, and also means we can support machines that have more than one sensor installed, for instance laptops with an inexpensive internal color sensor and an external expensive ColorMunki.

So, all we have to do to take advantage of this is write the 100 lines of GDBus code in gnome-control-center and the few lines of QtDBus code for the color KCM for KDE.

So where do we go from here? I’ve still got to polish up the colord-session code and do a first release. Then it needs to be added to jhbuild and Fedora 19. Then I need to patch gnome-control-center to use the new service for displays rather than calling out to the old gcm-calibrate helper. Daniel has to do the same for colord-kde.

Color calibration in GNOME 3.8 has just got sexy.

Color Management Hackfest 2012

Well, it was an excellent weekend for me.

General notes:

  • Till and I did a lot of testing of Ghostscript and found a very nasty bug indeed.
  • Daniel got arrested at customs, and any help for him would be most appreciated.
  • Øyvind (pippin) is always super interesting to talk to, and we discussed a way to do screen calibration with just a webcam in a new way. He also analysed all the VCGT data in the icc profiles from Taxi so that he could find a colour-managed resistant dithering routine to reduce the amount of banding in the default GNOME 3 wallpapers.
  • Chris Murphy (Color Remedies guy) is frickin’ awesome. For it not Chris coming to the hackfest I don’t think I would have learned half as much as I did. Chris and I basically talked for hours at an end about all the mistakes in OSX for color management, and my perception of the competency of the competition is much dented. He did some testing with GIMP and lcms and basically found we were doing the right things, although he uncovered a bug due to SELinux in Fedora 18 (fixed yesterday by Dan).
  • John Layt is the Qt print dialog dude. He was adding colord support to the Qt print dialog like we did for Gtk+. He isn’t happy adding dependencies for additional libraries but the raw DBus API from colord was exactly what he needed as he could do a soft-runtime dep.
  • Daniel Jahre has taken over the taxi database maintenance and him and Sirko are okay with my changes to support the SHA1 hash feature I need for GNOME.
  • Lukáš Tinkl fixed a load of UI and functionality bugs in colord-kde.
  • We all talked a bit about wayland and where to introduce different bits of the CM stack into the wayland model. It turns out we can do thing like we want without too many problems. The sticky problem of area-opt-out was discussed, but pippins idea was to have the opt-out-region as a sub-window in the window.
  • Kai-Uwe discussed the CM print path quite a lot. Most people concluded that libcmpx wasn’t the way to go. Him and Till did a lot of work about getting PDFs so that we could test the output intents and input source profiles on real printers. We discussed a lot about google cloud print too. Chris said that colord also lets us blacklist printers that lie and claim they support an output intent (or PDF/X) but really don’t.
  • I sat down with Chris and worked out with him how we can do a real print preview, without all the silly options that OSX has, for instance “Simulate black ink“. I coded a small print preview example that basically shows the user pretty much what they are going to get on paper assuming they have a printer profile and a display profile. Basically, it makes the image match what you see on paper, but also makes it look rubbish :) We’ll use this in the new GNOME print dialog.  In doing so we found an LCMS2 bug that I still have to debug and fix but that can be worked around. Pippin and I also debugged a cairo regression that means we have to leak a reference in GTK+ to avoid crashing when doing a print preview. I’ve reported it and done a tiny test case.
Print preview example (for SNAP-TR002)
So, things that I want to do in the next few weeks:
  • Start the pdftopdf outputintent work, after the ghostscript bug has been fixed (otherwise I’ll break everybodies printing…)
  • Put FOGRA39L on the Fedora LiveCD not 27L as it’s a better default nowadays
  • Enable CM by default for GIMP and firefox at least in Fedora
  • Chase up the lcms2, cairo and ghostscript bugs
  • Talk to krh about subwindows in Wayland

So apart from Ania having to look after a 4 week old baby on her own (which was pretty stressful for her) it was an excellent trip. We’ve agreed to repeat the event next year, and also do a track at LGM this year. The amount of interest in Linux CM seems to be genuinely increasing and it makes sense to take advantage of the momentum that’s growing.

So, +1 from me. Thanks go to Kai-Uwe, jreznik, sirko and the others who did a really good job organising everything.

ColorHug devices in stock

Ania and I spend a good chunk of this weekend calibrating more ColorHug devices, the open-hardware colorimeter. A combination of me being unwell last week and the stress of a house extension (and a baby on the way) meant this batch has been far more delayed than we wanted it to be. Devices can be purchased on the website with no pre-order required as we’ve now worked through the backlog and have got stock spare.

If you’re interested in the latest things we’ve been working on have a look the FAQ entry about the new measurement modes available in the latest experimental firmware.

Thanks!

Offline OS Updates – Looking forward to GNOME 3.6

All weekend I’ve been hacking on PackageKit and systemd to be able to have a GNOME 3.6 user experience that looks like the new update mockups.

So what’s the plan?

  • gnome-settings-daemon will automatically prepare the transaction using PackageKit downloading all packages (either all, or just security updates) and deps which in turns creates a /var/lib/PackageKit/prepared-update file when ready [works]
  • If /var/lib/PackageKit/prepared-update exists then gnome-shell will show a “Restart and install updates” option, that if clicked will call pkexec pk-trigger-offline-update which creates /system-update and the session is rebooted [needs gnome-shell patch]
  • On next boot, if /system-update exists, then the systemd generator starts system-update.target which in turn starts packagekit-offline-update.service, which in turn makes PackageKit run the prepared update transaction. On error, /system-update is removed, and on success both /system-update and /var/lib/PackageKit/prepared-update are removed. [works]
  • Plymouth will show a package icon (or something) with a widget that fills up as the transactions are processed (0 to 100%) [needs-work]
  • Plymouth will show a message after the updates are applied like “Rebooting after installing updates” [needs-work]
  • Show a message at next boot if the offline update succeeded or failed [working-on-right-now]

So why bother with all this?

  • Installing updates while the session is running causes havoc with some apps like firefox that have file resources that have not been locked (just try updating xulrunner when firefox or thunderbird is open…)
  • Installing library updates when apps are running against the old copies means the processed need to be restarted (gnome-session, sshd, etc) before the changes are in effect (for all users logged into the machine)
  • Installing core OS updates and doing OS upgrades in the running session works for most people most of the time, and then when it fails it destroys your system completely with no way to recover
  • Using a minimal pre-boot environment we can snapshot the system before we update the OS and afterwards (requires btrfs or something else)
  • Using a fresh pre-boot environment means we can easily check OS sanity before we start updating core bits of the OS, without lots of additional processes running.

Of course, we’ll still support updating applications in the session for GNOME 3.6 (as long as they are not running) just not the core OS bits. Comments, as always, welcome.

Linux Color Management Hackfest

From the 16th to 19th November we’re planning a hackfest in Brno, Czech Republic. The venue is kindly being sponsored by Red Hat.

What we’re trying to achieve is to get all color-minded people in the same place at the same time, to try to join up some of the color management stack in Linux. We’ll be discussing toolkit color management, the print color pipeline using CUPS and printerd and application level color management. It’s being arranged by S.Kemter and Kai-Uwe Behrmann so if you want to be there, email one of them or jump into the #openicc channel on freenode. There’s some sponsorship available, but not a lot. It’s also going to be a hackfest, i.e. writing code rather than giving talks like “a dummies guide color management”. I’m pretty excited.

Getting the ICC display profile

I’m at LGM this year, and so far it’s rocking pretty hard. The number one question people have asked is “how do I get the screen profile for a window“. I figured this should be easy to get using colord, and then spent a few minutes working on some proof-of-concept code. This ballooned into a couple of hours doing it properly asynchronously and making it work correctly on multihead, and the result was a few hundred lines of complicated code with quite a few exit points. I don’t want people to add 300 lines of boilerplate to their project just to map a GtkWindow to a .icc filename.

So I’m now shipping an additional optional colord-gtk helper library in colord that allow you to use one async function to get the profile a given widget should use. There’s a demo available here.

The alternative is of course to read the X11 _ICC_PROFILE atom, but that does not support multi-head, and really won’t work when we move to Wayland. It’s also not a lot of fun grabbing lots of binary data from the xserver in a GUI program. In the long term future we’ll be doing full screen color management in shaders, with full toolkit support using Wayland, but that’s a few years from being reality. If you’ve got any ideas or have comments about the API, let us know on the mailing list. Thanks.

Stop wasting time and money, make the Fedora 18 release name “Fedora 18”

Calling all Fedora users and developers. Please visit the official poll to choose the future of Fedora release names.

Nobody refers to “Running Fedora Verne” and choosing the name every few months is just a giant waste of time and waste of a very busy legal team that has to review and research each stupid name. I think “Beefy Miracle” is a ridiculous name that really takes the edge off an otherwise most professional release.

Just have the next release name as Fedora 18 and be done with the nonsense names once and for all.

colord-kde 0.2

Daniel released colord-kde 0.2 today. I’m really excited about the amount of progress he’s made in such a short amount of time. He’s has had a bit of rough ride with the loss of his daughter last year and this year he’s been hacking on colord-kde and PackageKit and trying to make open source a better place. If you’ve got a spare couple of dollars this month, he’d really appreciate a donation to buy hardware to test things on.

I was a strapped-for-cash geek myself a few years ago, and I know how hard it was to develop software when you don’t have the right hardware. I donated $25. Donate here. Thanks.