Oh what joy this whole GNU/Linux thing brings. I took a few days off to upgrade my machines. I had the pleasure to update one laptop twice, i.e. from the Ubuntu 12.04 LTS to the current 13.04 and a desktop from Fedora 17 to Fedora 18.
The Laptop was almost easy. It took long time for the system to install packages. And there are stupid dialogues to confirm which block the whole process. Not very nice. I let it run for a couple of hours, everything went more less fine until I couldn’t log in anymore. LightDM saved my GNOME preference but there was no gnome-session left. So I went to the console and got myself ubuntu-gnome-desktop (arr. stupid wordpress doesn’t render apt:// links).
The second update from 12.10 to 13.04 took as long as the first, with nothing noteworthy happening. Interestingly though, it didn’t want to install the 13.04 unless being told to install a “development release”. Bollocks.
Anyway, Ubuntu’s GNOME runs almost nicely on my tiny laptop. GNOME-Shell is very slow when it comes to alt-tab. It takes three or four seconds to switch a window. Distraction free computing at its best.
The Fedora desktop is full bucket of joy. The FedUp utility keeps what it promises. It’s surprisingly refreshing. This time, the whole upgrade procedure worked flawlessly. No really! In 2013! I’m amazed. It only took a while for it to fetch everything but then a reboot straight into the upgrade system made the magic happen. Very cool.
Not so cool was the surprise of the machine not booting. Of course. Systemd hung somewhere in NFS related daemons and bailed out because they failed. The old GRUB menu entry booted a little further, just until sendmail, and enabled me to investigate.
Sendmail could not be brought up, because “-bd is not supported by sSMTP”. Right. I have sSMTP installed. And to make a long story short, something did place an init script in /etc/rc.d/init.d/. And that script failed now. NOW. After a couple of years. It was probably never used but got activated with the migration to systemd. Anyway, you might want to delete your stray init scripts and eventually get rid of the packages altogether.
Then GDM wouldn’t come up. Only flicker. It took me a while to find the relevant log files (thinking that everything was in the Journal by now…) but grepping for the usual “EE” and “WW” didn’t reveal much.
# grep -r -e EE -e WW /var/log/gdm/
/var/log/gdm/:5.log.1: (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
/var/log/gdm/:5.log.1:Initializing built-in extension MIT-SCREEN-SAVER
/var/log/gdm/:5.log.1:(WW) Falling back to old probe method for vesa
/var/log/gdm/:5.log.1:(WW) Falling back to old probe method for modesetting
/var/log/gdm/:5.log.1:(WW) Falling back to old probe method for fbdev
/var/log/gdm/:5.log: (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
/var/log/gdm/:5.log:Initializing built-in extension MIT-SCREEN-SAVER
/var/log/gdm/:5.log:(WW) Falling back to old probe method for vesa
/var/log/gdm/:5.log:(WW) Falling back to old probe method for modesetting
/var/log/gdm/:5.log:(WW) Falling back to old probe method for fbdev
/var/log/gdm/:1.log.2: (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
But. There were also the logs for the “slaves”. They contained:
gdm-simple-slave[1030]: WARNING: Failed to give slave programs access to the display. Trying to proceed.
gdm-launch-environment][1046]: pam_unix(gdm-launch-environment:session): session opened for user gdm by (uid=0)
gdm-launch-environment][1046]: pam_unix(gdm-launch-environment:session): session closed for user gdm
gdm-simple-slave[1030]: GLib-GObject-CRITICAL: g_object_ref: assertion `object->ref_count > 0′ failed
gdm-simple-slave[1030]: GLib-GObject-CRITICAL: g_object_unref: assertion `object->ref_count > 0′ failed
And there was a hint given by systemd:
# systemctl status gdm --full
gdm.service - GNOME Display Manager
Loaded: loaded (/usr/lib/systemd/system/gdm.service; enabled)
Active: active (running) since Fr 2013-05-03 12:22:04 CEST; 9s ago
Main PID: 1843 (gdm-binary)
CGroup: name=systemd:/system/gdm.service
└─1843 /usr/sbin/gdm-binary
Mai 03 12:22:07 bigbox gdm[1843]: gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.510350 seconds
Mai 03 12:22:07 bigbox gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.510350 seconds
Mai 03 12:22:07 bigbox gdm-simple-slave[1997]: WARNING: Failed to give slave programs access to the display. Trying to proceed.
Mai 03 12:22:08 bigbox gdm-simple-slave[1997]: GLib-GObject-CRITICAL: g_object_ref: assertion `object->ref_count > 0' failed
Mai 03 12:22:08 bigbox gdm[1843]: gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.507905 seconds
Mai 03 12:22:08 bigbox gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.507905 seconds
Mai 03 12:22:08 bigbox gdm-binary[1843]: WARNING: GdmLocalDisplayFactory: maximum number of X display failures reached: check X server log for errors
Mai 03 12:22:08 bigbox gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.509609 seconds
Mai 03 12:22:08 bigbox gdm[1843]: gdm-binary[1843]: WARNING: GdmLocalDisplayFactory: maximum number of X display failures reached: check X server log for errors
Mai 03 12:22:08 bigbox gdm[1843]: gdm-binary[1843]: WARNING: GdmDisplay: display lasted 0.509609 seconds
Aha! There is the problem! But.. what is it? No indication whatsoever. Not even a tiny hint as to where to look next.
I decided to make baby steps and tried to bring up X on my own. My computer liked “X”. But it didn’t “startx”. That in turn revealed a missing library. libicule.so.48. But the current version is .49. Why on earth would something try to link against an old version? “yum distro-sync” proves me right that my packages are up to date. I thus set out to find the weird library causing me trouble. But there were many!
# ldd /lib64/libgailutil-3.so | grep not
libicule.so.48 => not found
libicuuc.so.48 => not found
libicudata.so.48 => not found
I thought I got rid of them by doing
for f in /lib64/*.so; do ldd $f | (grep -q “not found” && echo $f); done | xargs yum remove -y
but that didn’t help. The ldd resolves symbols recursively but I really want to know the symbols needed by the library itself, not its dependencies. Readelf comes to mind. And after chasing a few libraries manually, I was tired so I came up with
for lib in $(cat /tmp/libs); do echo $lib; for l in $(readelf -d /lib64/$lib | grep NEEDED | cut -d[ -f2 | cut -d] -f1); do echo $lib: $l; done; done | less
which showed nicely which library the culprit was.
It was /lib64/libharfbuzz.so.0 from harfbuzz-0.9.13-1.fc20.x86_64. Where does this package come from, you may ask. So did I. I didn’t know how to make yum tell me, but I found out that it belonged to the F17 texlive repository.
Interestingly enough, yum check told me that there was a problem but couldn’t handle it. The solution, very similar to the command above, but with an important difference:
yum --disablerepo texlive distro-sync
Hope this will be useful to someone in the future. Chances are quite good.
