Upgrade log

Random bits from the menubar upgrade earlier today:

• Upgrade started at 10:30 UTC when Matthew Galgoci upgraded the machine from RHEL3 to RHEL5. I think it was the middle of the night for him, did not dare to ask though. This took in total about 2 hours. A long part of that was spend on adding SELinux labels.
• Stuff that broke in some way:
  • website (needed to change config to include GNOME bits)
  • postfix (due to: being disabled on purpose, RHEL5 installing sendmail and forced downgrade to the RHEL5 postfix instead of the newer custom postfix rpm it still had from RHEL3)
  • NFS (it didn’t install nfs-utils for some reason)
  • bind (error in a zone file that the previous Bind did not care about)
  • amavisd (due to *much* newer clamav. this used a different config file; that config file didn’t enable the localsocket plus socket location changed)
  • clamav (didn’t have access to /var/spool/amavis)
  • mailgraph (the custom rpm used different file locations than the custom rpm used for RHEL3)
  • script around mhonarc (due to overlooking the perl-Mail-Field-Received custom rpm that needed a rebuild)
  • p0f (worked after I restarted it.. not sure why)
  • mailman (didn’t break, just rebuild it because of the newer Python)
• Menubar sent at one point around 1850 mails/minute
• Menubar received at one point around 2400 mails/minute
  
• Unfortunately, above numbers are much lower than what menubar can really do. This as menubar still had 3 config problems at that time; causing it to be much slower than usual. Problems in short: p0f running at 100%, hung mhonarc process slowing down mailman archive processing and amavis checking every bugmail.
• Found a yum plugin that works like protectbase, but for RHN. This works really nice in combination with custom YUM sources (Dag Wieers). Meaning: Only upgrade stuff from other sources if RHEL5 doesn’t provide it. This as I trust RHEL5 more (purpose of custom yum sources is usually to get the latest version asap.. but those newer versions can have bugs in them.. or work differently than before).
• Changed the timezone of the server to UTC
• Mail server config was tested by: firewalling port 25 externally, binding postfix to localhost and sending test mails to bugmaster until every problem was fixed (except the slow mailman processing.. caused by hung mhonarc process.. I noticed that early on, but overlooked the 2nd mhonarc process). After that I first enabled all interfaces (while still being firewalling port 25 externally) to let menubar process the bugmail queue (plus Wiki mails). This is a good stress test. After that I opened port 25 externally and flushed the mailqueue on socket (causing lots of svn-commits-list emails in a very short time). Only after most of the svn-commits-list queue was processed did I notice the cause of the slow mailman.. and p0f using 100% CPU (oh well).
• Upgrade as good as completed at 16:45 UTC (external mail worked already at ~15:00). The machine still needs a reboot (newer kernel).

and with above another item of the 2.22 RoadMap has been completed.