I think it’s been a little while since my last post, so I thought I’d toss something out there to our loyal readers (all three of you!). What have we been working on this week? Let’s see…

I pushed hard (maybe too hard) to get the new servers out of my house. The Foundation purchased two new servers and we’ve been trying to coordinate their installation for quite some time now. It finally looks like we have the correct address, contacts and coordination to get them put in. I should be able to, fingers crossed, get them out this weekend.

Let’s see, previous to that we had the big GNOME 3.0 release which overall went well. It put some pressure on some places, but that turned to be a good thing because we found where we could improve. Each time we’re pushed we learn one more thing about how we can streamline and improve our setup. It’s a good challenge, if anything.

I’m happy to say, after the initial tweaking, we stood up to slashdot which you might imagine is not a simple feat for many sites.

Ohh, a few more somewhat sysadmin related announcements have been in regards to the snowy project. If you’re not familiar with Snowy, think “Ubuntu One” but open source and specific to Tomboy. You can find out more here and here.

For now that’s all I’ve got. Signing off.

As you may have read, we’ve finally made the move to WP 3.1 and migrated from the WPMU installation to the integrated multi-site support in the mainline WordPress latest release. This happened yesterday afternoon, and other than some DNS propagation we didn’t anticipate any problems.

Well, it turns out that some of the caching plugins and configuration didn’t properly migrate / upgrade / function after the move so the server went down under heavy load over night. I spent some time this morning tweaking and monitoring, and tweaking again until I felt like the right caching solution was in place. It should now be applied to all blogs within the multi-site network, and should improve performance significantly.

During this monitoring and tweaking I was closely watching the error_log output for anything that might also need attention. There was one other thing that I found that needed attention. Outdated and unmaintained WordPress themes that were causing PHP Parse errors.

I spent some time trying to find upgraded versions for the themes, and updated where I could. There were three themes that I was unable to find updated versions for and so it was decided to clobber them entirely.

If you’ve found that your blog has suddenly reverted to the default theme, it is likely that you were using one of the outdated themes that had to be removed. My apologies. Please feel free to select a different theme or suggest a new theme for installation.

As usual, please accept our apologies for these service disruptions.

Here I am sneaking in some last minute updates before the end of the year. I know I just posted an update a few days ago, but I’ve implemented some additional improvements since then that I wanted to share.

First of all, and this is pretty cool I think, we’ve implemented HTTP Strict Transport Security for all GNOME domains that require SSL. For the end-user this primarily means bugzilla and tomboy-online. If you’re not familiar with HTTP Strict Transport Security, check out the spec here. Essentially what it does is has the web server send a specific header ‘Strict-Transport-Security’ to the client browser, including a TTL. This header value is stored and used by supported browsers (Chrome and Firefox 4 currently) which force the browser to connect directly on https on the next connection (more specifically, during the TTL time value). This means that once you’ve visited bugzilla and connected to it via SSL, any future connections inside that TTL time value will go directly to the SSL site and never touch the non-SSL. (Normally connections not explicitly connecting to https are redirected from http to https.)

I think this is a pretty cool security addition, and closes another sysadmin ticket!

Second, my main project for the day has been our monitoring system. It got some attention today by way of adding a few missed hosts as well as adding some remote checks for services. We’re now tracking the health of most of our mysql servers and watching a few new values such as load average, etc. I’ve also added checks to monitor the status of our SSL certificates. Currently these checks will start to alert us when the certificate is within two weeks of expiration.

Ohh, I also added a certificate to mail.gnome.org which previously didn’t have one. This now allows (but doesn’t yet require) secure connection to the mailing lists, archives and administration pages. I felt this was important given that credentials are passed between client and server for mailman list administration and queue management, which is another task myself and the Moderators team handle daily.

I think we’ve made some good progress this week and hopefully it sets a good precedent for the new year.

Christer

Just before the end of the year I thought I’d give one last status update on what I and the team have been working on. As you may imagine, things have been slow due to the holiday season, but we’re still here and still keeping the gears moving.

The most recent success has been improving and fixing the monitoring solution, Nagios. Just last night I finally properly implemented SSL for Nagios administration logins and setup redirects from the old URLs. Currently Nagios is admin-only, but I am considering the idea of a public view so that the GNOME community can get a glance at what we monitor and can check the status of services and hosts.

Some of the other tasks we’re working on is preparing to migrate some hosts to RHEL6. We’ve got the RHEL6 images imported into our build system, but unfortunately we’re stuck on a networking issue in the automated installation. I think once the whole team is back from holiday we’ll get it figured out and have some built. The first boxes on the list for RHEL6 are the wiki, snowy and blogs.gnome.org.

We made a recent update to the mail server filter to remove the SORBS rbl. They’ve changed their policies and added some questionable address ranges which have caused us some problems. They’ve been replaced with a different rbl.

We’re also looking to consolidate the GNOME related domains into a single registrar. We’re shopping around for a dependable, free software friendly registrar. If you have any suggestions, please comment or contact the team. We’re very interested in input regarding where our domains can call home.

Beyond this I’ve mainly done general administration and maintenance. Mailing list queue moderation, account updates and creation, had to take care of a corrupt table in the piwik database.. you know, the general day to day stuff. As of today I’ve clocked about 33hrs for the month. I hope, during this last final stretch before the new year, to add ten more hours to that number and really tackle some more bugs.

As usual, if you have any questions or comments for the team please let us know. We’re happy to help, we just ask that you communicate and follow-up with us with any issues you have.

Christer