GResource and colord startup

A couple of days ago I released colord 0.1.30. This was an otherwise unremarkable release with the normal splattering of a few bugfixes and the occasional small new features.

One such feature is the use of GResource. The new GResource stuff that landed in Glib 2.32 allows you to embed abritary binary data into the actual executable file. This is typically used for embedding small files that are normally loaded at runtime, for instance D-Bus introspection files or small application icons. Embedding that data also lets us strip the blanks from any XML file, and optionally compress the data too. It means we’re not seeking and loading many small files when the binary is run, which reduces by a small amount the amount of I/O that is done, and hence speeds up startup.

So I got thinking. Looking at the cold startup[1] I/O profile of colord, the first thing it does is scan for any files in /usr/share/color/icc for *.icc files. On the default system in Fedora, we only have a few files installed in that directory, and all of them are generated by colord at package build time and shipped in the colord package. We know where they are going to be, and what the contents are. Typically there are ~10 profiles installed, and they are all less than 1kb in size.

Since 0.1.30, at build time the profiles generated by colord (and only those) get included into the binary as resources. This means the colord binary size grows by slightly less than 10k, but means we don’t load 10 small files from the disk at startup. The files are still installed like normal so that applications can reference them as files like before, but if there is an internal mmap’ed copy of the same profile we use that instead. This reduces the amount of I/O that colord does at startup by about half. It speeds the daemon startup by about 35ms on SSD hardware (as seeks are cheap) but on spinning rust drives or LiveCD media it makes an order of magnitude more difference.

[1] cold, as in not in hot-cache. Do ‘echo 3 > /proc/sys/vm/drop_caches’ if you want to see the difference.

 

Color Calibration Survey Results

A couple of weeks ago I asked people on my blog and a few chosen mailing lists to answer three simple questions:

  1. What monitor calibration devices do you own?
  2. Which of these devices have you used in the last 6 months?
  3. If you were to buy a new calibration device, which would you buy?

I wanted to work out what hardware I should buy for testing with gnome-color-manager and colord for each release. The results are very skewed toward Linux users, but that was kind of the point of the survey.

So, the first set of data, which 203 people answered:

Notable points:

  • Nobody owns a Colorimetre HCFR. Not much of a suprise really.
  • Spyder4 is new hardware which performs well, but hardly anyone owns one yet.
  • 43% of people answering the survey own a ColorHug, which isn’t too much of a suprise since it was posted on the ColorHug Google+ page. Still, pretty awesome for such a new project.

The next graph is very similar to the first, with 191 people responding:

Notable points:

  • There are a lot of Spyder2’s sitting in drawers unused.
  • Lots of people bought a ColorHug, and don’t use it very often. This isn’t suprising as it’s the least expensive device by a long way.
  • i1Pro owners use the device a lot more than people that own other devices. This is also a very expensive device, so again, kinda makes sense.

The last graph is interesting in a number of ways, from 173 users:

  • 52% of Linux users would buy a ColorHug. This is the most popular cheap colorimeter option.
  • 31% of people would buy a much more expensive photospectrometer rather than a colorimeter.
  • 11% of people want to buy hardware considered obsolete by the manufacturer.
  • 2 people want to buy a Colorimetre HCFR. Good luck there :)
  • 6 people wanted to buy a ColorHug Spectro, even though it wasn’t an option on the survey and doesn’t even exist yet.

Based on the results of this data, I think it’s important for me to buy some Spyder hardware and concentrate on the photospectrometer-type hardware. Thanks to all respondents, your help has been really valuable.

 

Color management in GNOME 3.8

I’ve spent a few hours each day for the last couple of weeks writing code to support the new mockups done by Allan for display calibration. This involved two pretty big patches, one for a reworked color panel in the control center, and one for the colord-session native calibration I blogged about a few weeks ago.

I’m glad to say this landed upstream yesterday after review by Bastien, and now people are trying it out and finding niggles which I’ve been busily fixing.

There are lots of nice features which required adding quite a few new properties to colord, the daemon which makes all this UI possible. colord now knows (from the kernel) if a device is internal, and can’t be removed. This allows us to make a few of the translations much better. We’ve also got the ability to “turn off” color management for a device, which is persistent between reboots. We can also remove profiles automatically added by colord (using metadata information) which is also stored in a database.

To test this, you currently need colord, gnome-settings-daemon and gnome-control-center from git master, although we’ll be doing some tarball releases in the next few days.

Logitech Unifying Devices

Logitech have started shipping Unified devices, which means you can have up to 6 wireless devices connected to one little USB dongle thing. They’ve also invented a protocol called HID++, and it’s already in version 2. It’s basically a way to issue low level commands in a structured way to HID devices. It’s actually tons better than Bluetooth as it’s got a lower latency and also uses up less power on the sender and receiver.

I’m fairly familiar with low level protocols like this, as this is just the kind of thing the ColorHug is doing. Write a request packet, and get a response packet a few ms later. Julien Danjou had already added some support for the Unifying devices to UPower, albeit just for the K750 keyboard. Armed with a draft HID++v2 specification document and a newly purchased T620 I figured I could support most devices very easily by implementing the Battery Unified Level Status part of the specification.

Tuesday evening I started reading the specification sheet. By Wednesday lunchtime I had some test code in UPower that should have worked for all devices. But it didn’t. The percentage value was always 4%.

So, back to my trusty Windows XP VM. I loaded the Logitech client software, and that reported the percentage value correctly. I then dug out a hardware USB protocol analyser and made a trace of what the Windows client software was doing. Bingo. The BatteryLevelStatus ASE/SWID nibbles were swapped in the request packet. Either the documentation is wrong (unlikely, given the other ASE’s are the right way around) or the Logitech firmware engineer implementing the specification got them the wrong way around. I can’t blame them, the specification doesn’t explicitly specify an order, although you could easily work out what is likely as the byte order is specified as big endian.

I’ve pushed my patch to UPower. Testing very welcome.

Introducing display calibration using colord-session

This year I never got around to using my annual holiday. To use up some of the time that I cannot transfer over to next year I took the last 4 days off work to put up Christmas lights, buy a Christmas tree and create a new bit of shared infrastructure for calibrating displays.

I’ll describe how we calibrated displays in GNOME 3.4 and GNOME 3.6:

  • User opens the color control panel and clicks on the device, then “Calibrate”
  • We spawn the gcm-calibrate executable with the colord device-id as an argument
  • The user goes through a GTK wizard, and we calibrate the display to a target whitepoint and then characterize the display.

To do the last bullet point involved spawning dispcal and dispwin in a VTE widget and screen-scraping the output from ArgyllCMS. This would break every release and was really fragile. Also, dispwin and dispcal would show it’s own GL window with the sample window, so we’d have bugs where the window was on the wrong output (as argyllcms sometimes didn’t match the xorg display numbering) and also where the sample window was covering up the instructions in the GtkWizard. And if either of the tools crashed or the sensor was removed before finishing, then undefined things happened, ranging from a crash to a disastrous hang where the user couldn’t even click the Quit button. And to top it all off, there was no sensible progress completion support, so the user basically didn’t know if the calibration was taking a long time, had crashed or was waiting for input in the (hidden) VTE widget.

Basically, it was a hack, and just not good enough.

KDE did something similar. So much so, it used the same gcm-calibrate binary. This meant you switched from the native KCM in KDE and were popped into a GtkWizard with the same separate GL window issues. This meant you had to install quite a few of the GNOME libraries just to be able to calibrate your display in KDE.

So, not good enough, squared.

So I did three things in the last week or so:

  • Added a “spotread” sensor type in colord. This sandboxes the argyllcms spotread process and abstracts out any undesirable behavior. You can read more about it here.
  • Added a CdSampleWidget object to libcolord-gtk for simplicity (e.g. cursor hidden at the right time)
  •  Created a colord-session D-Bus activated process (that only gets started when a display gets calibrated and quits as soon as the caller quits or calibration finishes). This does all the heavy lifting in the session.

The latter has signals like ::UpdateSample() to update the color sample on the screen and ::UpdateGamma() to update the 2D LUTs which allows a small D-Bus wrapper to show any kind of GUI it likes. This lets us implement the new GNOME design mockups designed by Allan which look a lot less scary than the old UI. We can also pass a device-id and sensor-id to the new Start() method which means the calibration window gets shown on the right screen 100% of the time, and also means we can support machines that have more than one sensor installed, for instance laptops with an inexpensive internal color sensor and an external expensive ColorMunki.

So, all we have to do to take advantage of this is write the 100 lines of GDBus code in gnome-control-center and the few lines of QtDBus code for the color KCM for KDE.

So where do we go from here? I’ve still got to polish up the colord-session code and do a first release. Then it needs to be added to jhbuild and Fedora 19. Then I need to patch gnome-control-center to use the new service for displays rather than calling out to the old gcm-calibrate helper. Daniel has to do the same for colord-kde.

Color calibration in GNOME 3.8 has just got sexy.

Color Management Hackfest 2012

Well, it was an excellent weekend for me.

General notes:

  • Till and I did a lot of testing of Ghostscript and found a very nasty bug indeed.
  • Daniel got arrested at customs, and any help for him would be most appreciated.
  • Øyvind (pippin) is always super interesting to talk to, and we discussed a way to do screen calibration with just a webcam in a new way. He also analysed all the VCGT data in the icc profiles from Taxi so that he could find a colour-managed resistant dithering routine to reduce the amount of banding in the default GNOME 3 wallpapers.
  • Chris Murphy (Color Remedies guy) is frickin’ awesome. For it not Chris coming to the hackfest I don’t think I would have learned half as much as I did. Chris and I basically talked for hours at an end about all the mistakes in OSX for color management, and my perception of the competency of the competition is much dented. He did some testing with GIMP and lcms and basically found we were doing the right things, although he uncovered a bug due to SELinux in Fedora 18 (fixed yesterday by Dan).
  • John Layt is the Qt print dialog dude. He was adding colord support to the Qt print dialog like we did for Gtk+. He isn’t happy adding dependencies for additional libraries but the raw DBus API from colord was exactly what he needed as he could do a soft-runtime dep.
  • Daniel Jahre has taken over the taxi database maintenance and him and Sirko are okay with my changes to support the SHA1 hash feature I need for GNOME.
  • Lukáš Tinkl fixed a load of UI and functionality bugs in colord-kde.
  • We all talked a bit about wayland and where to introduce different bits of the CM stack into the wayland model. It turns out we can do thing like we want without too many problems. The sticky problem of area-opt-out was discussed, but pippins idea was to have the opt-out-region as a sub-window in the window.
  • Kai-Uwe discussed the CM print path quite a lot. Most people concluded that libcmpx wasn’t the way to go. Him and Till did a lot of work about getting PDFs so that we could test the output intents and input source profiles on real printers. We discussed a lot about google cloud print too. Chris said that colord also lets us blacklist printers that lie and claim they support an output intent (or PDF/X) but really don’t.
  • I sat down with Chris and worked out with him how we can do a real print preview, without all the silly options that OSX has, for instance “Simulate black ink“. I coded a small print preview example that basically shows the user pretty much what they are going to get on paper assuming they have a printer profile and a display profile. Basically, it makes the image match what you see on paper, but also makes it look rubbish :) We’ll use this in the new GNOME print dialog.  In doing so we found an LCMS2 bug that I still have to debug and fix but that can be worked around. Pippin and I also debugged a cairo regression that means we have to leak a reference in GTK+ to avoid crashing when doing a print preview. I’ve reported it and done a tiny test case.
Print preview example (for SNAP-TR002)
So, things that I want to do in the next few weeks:
  • Start the pdftopdf outputintent work, after the ghostscript bug has been fixed (otherwise I’ll break everybodies printing…)
  • Put FOGRA39L on the Fedora LiveCD not 27L as it’s a better default nowadays
  • Enable CM by default for GIMP and firefox at least in Fedora
  • Chase up the lcms2, cairo and ghostscript bugs
  • Talk to krh about subwindows in Wayland

So apart from Ania having to look after a 4 week old baby on her own (which was pretty stressful for her) it was an excellent trip. We’ve agreed to repeat the event next year, and also do a track at LGM this year. The amount of interest in Linux CM seems to be genuinely increasing and it makes sense to take advantage of the momentum that’s growing.

So, +1 from me. Thanks go to Kai-Uwe, jreznik, sirko and the others who did a really good job organising everything.