Nazgul on GNOME Bugzilla

Christian Kirbach will now handle all product creation requests on GNOME Bugzilla. Hopefully he likes perl as well… for his sake.

In other news, about 1500 people clicked on the porno spam link posted as a few HTML files in GNOME Bugzilla. Spammer has been banned of course, plus the files aren’t visible anymore. Please inform bugmaster@gnome.org, or someone in #bugs on irc.gnome.org when you notice this in future.
Aside from GNOME Bugzilla, the spammer abused at least Red Hat Bugzilla, and some wiki site. The spammer didn’t use a bot btw, the account creation was manual (easy to determine from the logs).

Read (via Digg) a blog about 14 rules for fast web pages. Enabled that gzip stuff for Bugzilla. Ignored the rest.

Slow ticket processing

I was questioned by Vincent at UDS if the GNOME sysadmin team needed help as some tickets took a while to process. We (me,Vincent) didn’t have the time to discuss it, but I mailed my thoughts to him. He suggested I’d blog about it. This is it. Below is the almost pristine mail that he received. He did suggest I’d make a summary, but it is late and I do not want to. I slightly reworded small parts though.
PS: These are my thoughts and plans. Haven’t discussed this on gnome-sysadmin.

There are a few problems:

  • Account requests take long to process
  • Mailing list requests take long to setup correctly

I am not aware of problems aside from above (meaning, AFAIK everything is running smoothly). Note that not everything
is sysadmin related. E.g. I am *very* slow with Bugzilla related bugs
(‘tickets’) currently due to lack of time. However, that is not a
sysadmin problem.

Long to process account requests
First of all, this is handled by the accounts team; not by the
sysadmin team. Although in practice there is an overlap. In short:
some sysadmins do not setup accounts; and some accounts people are not a sysadmin (have root). The delay is not just new accounts, also e.g.
updating the email address takes a long time.
I think this is basically caused by the following:

  • Some of the active accounts team have been scared away (some
    requests are so ‘friendly’ that a few accounts@ people quit).

  • Some people are just busy with other stuff
  • Handling accounts is a stupid boring task that should be automated for 99%; it is not fun
    nor rewarding (people have better things to do with their time)

Mailing lists setup takes a long time
I refuse to setup new mailing lists as it will only result in a broken
one. Nobody else seems to handle these within a reasonable timeframe,
so the end result is that they take months to process.

Solutions to the problems I stated initially that will not work:

  • Add people to the sysadmin team
  • Add people to the accounts team

Add more people to sysadmin team
From my experience I noticed that adding more people really
does not help (apart from making sure they can be trusted, are a
sysadmin, etc). The people who have the new privileges usually stop
helping after a few months at most. Although people do want to help,
it seems they don’t keep the interest.

Someone I would add is someone who continually says what needs to be
changed and how to do it (in a constructive way); plus knows what
should not be done. The reason for this is that as a sysadmin there is
not someone who will tell you either what to do or how to do it.
Meaning, a sysadmin will need to know that by heart.
In short: I want people who would join anyway, not people who want to help.

Add people to the accounts team
Due to reasons I specified before, I think it is better to automate
much of the task away (Summer of Code project) than spend lots of time to find more people to
do a boring task. Although I’d appreciate people helping out with
enhancing Mango.

Other causes:

  • Not every sysadmin follows RT3

Not every sysadmin follows RT3
Not sure why some do not have an account or they do not look at
tickets. They are subscribed to gnome-sysadmin though (which mostly receives cron/logwatch, etc etc).

I hope most problems can be solved by the following:

  • Implementation of new account creation system within mango
  • Redo the mailing list setup.
  • Avoid using RT3, make use of Bugzilla instead (sysadmin product).

New account creation system
This should provide transparency and automate most of the process.
This hopefully also includes ways to securely have people change their
account as well (e.g. add/remove SSH keys, update email address, etc).

Redo mailing list setup
Basically the mailing list setup is so strange that when a new mailing
list is created, it fails to work. Usually either the alias does not
exist (you cannot mail to it), the archive is not listed or the
archive does not work. I plan to redo the mailing list setup together
with the upgrade of that machine from RHEL3 to RHEL5.
Note: Currently it feels to me like a black box. Symlinks to symlinks,
etc. Also the available sysadmin instructions do not result in a
mailing list that actually works. This is also the reason the machine
still runs RHEL3 (I do not want to break it).
Oh, and the indexing of the archives is still broken.

Avoid using RT3
We currently require RT3 basically for the account system and generic
requests. This because we need a system which can mail random people
to setup accounts. Tickets are not shown as we have no way to verify
someone did not change the From:. The solution is sending people a
token and verifying that they mail this token back.
For the non-accounts uses I think we should be able to use Bugzilla in
theory. However, currently something that I do not want to explain publicly blocks a switch from RT3 to Bugzilla.
Note: gnome-sysadmin@gnome.org will stay for urgent requests and/or
when e.g. Bugzilla is down (which would be an urgent request).

Bugzilla 3.0

Part of this was copy/pasted from the announcement.

Here’s just a sampling of the major new features in version 3.0:

  • Less ugly perl code (almost everything is readable&understandable now)
  • Custom Fields
  • mod_perl support for greatly-improved performance
    While sucking up loads of memory 🙁

  • Per-Product Permissions
    non hacky version of what is available on GNOME Bugzilla (patch wasn’t by me)

  • XML-RPC Interface; see the WebService stuff on this page
  • Create and Modify Bugs by Email
    Still not GPG authenticated though; so I won’t enable it on b.g.o without at least a hack to make it more secure.

  • And even more. See all the new features at: http://www.bugzilla.org/releases/3.0/new-features.html

GNOME Bugzilla
I am really busy. Help with the port is appreciated. The current (very rough and unfinished) port is available in the bugzilla.gnome.org SVN module; while the site currently uses the bugzilla-newer SVN module. I think that probably the port should use Bugzilla HEAD (what will eventually be 3.2) though. I basically need at least 3.0 to easily avoid those bug-buddy crash duplicates.

Name for the build-brigade machine

In the near future I’m going to setup the build-brigade machine. This machine is currently known as tinderbox1. As we’ll be using buildbot, that machine name is wrong (the ‘1’ also isn’t nice). I could just name it buildbot, but ideally a more generic name would be nice.

Suggestions?

Please add comments to the blog or as a reply to this mail (just mail build-brigade-list@gnome.org)

Library.gnome.org alpha 2

For those not following #docs on IRC and/or the gnome-web-list mailing list, here is another status update on the work done by various people on library.gnome.org.

Things changed:

  • gnome-doc-utils documents look more like the design by Frederic Peters (created from an image)
  • libgo now creates index.html.$LANG files for all the gnome-doc-utils documents.
  • Uses the new gnome-doc-utils HTML output instead of XHTML (you cannot serve XHTML as HTML.. unless you make it pretend to be HTML, but IMO then you should just use HTML anyway)
  • Changed a bunch of Python code to remove UNIXisms (and introduced a few bugs..)

Things that are broken:

  • Some modules have hacks in their Makefile.am files, making it impossible for libgo to find the documentation
  • Sometimes libgo can’t find the figures (often for gnome-doc-utils)
  • gtk-doc often have missing xml files
  • gtk-doc are not indexed (for now just browse http://library.gnome.org/api)
  • Feature: separate user and developer documentation.
  • Instead of calling it library.gnome.org everywhere, perhaps ‘GNOME Library’
  • Feature: instead of one big unreadable logfile, have some kind of structured output

Library.gnome.org design

We just had a first meeting in #docs about library.g.o. Frederic Peters announced a very nice library.g.o design on his library.g.o testsite. Screenshot:

This doesn’t build yet. For this we need a new gnome-doc-utils. The first test version of that is ready. Now the libgo software needs to be enhanced to make use of the new gnome-doc-utils possibilities (toc in the sidebar). A screenshot of what the new gnome-doc-utils can do:

I’ve updated the server with the gnome-doc-utils out of SVN. This means library.g.o is broken until someone updates the XSLT in the libgo software.

Mango
Lately I’ve been fixing a few things that annoyed me in mango (system used for handling the GNOME LDAP accounts):

  • HTML pages didn’t use UTF-8 everywhere.
  • Couldn’t add more than one SSH key. The other ones where silently deleted. This means those tickets could only be handled by a sysadmin, and not any accounts@ person. Plus that sysadmin had to know how to do LDAP via the commandline.
  • Sending a standard mail from Mango to someone with a dot in the name gave an error message
  • Emails weren’t in UTF-8
  • Login box wasn’t automatically focussed
  • Some of the standard email templates weren’t in mango

Above are mostly small fixes. The most annoying thing is that we can’t create UTF-8 accounts. I think this is because the openldap scheme by default doesn’t allow UTF-8 in the gecos field. However, I am not sure if I can just change that and not have it break the existing LDAP entries. Further, it shouldn’t break when e.g. a security rpm for openldap is installed.

Mango revamp (Summer of Code project)
Baris Cicek is working on enhancing mango (the system used to handle the GNOME LDAP accounts). The idea is to make the account requesting much faster and more transparent. The current ideas are listed on http://live.gnome.org/Sysadmin/Mango_Revamp (feel free to read and add comments — only at the end though!). The current method basically sucks. Ideally I’d like accounts@ to consist of only checking the new user and click either a ‘create account’ / ‘reject’ button. Before that the maintainer already approved everything (securely); the email address was verified to be correct, etc. There are some difficulties with designing the system, a few of them are:

  • No list of maintainers for the different modules. There are MAINTAINERS files, which is often used as a free text field. Moreover, the email addresses often do not match. Further, I rather have LDAP have an official list than have mango rely on whatever system we use for the source code (SVN). But on the other hand, I guess maintainers would like a MAINTAINERS file more than some webinterface (guessing).
  • No good way to authenticate. Sending an email with a token is not secure enough. A password is worse, when that is lost the whole SSH publickeys breaks down. I’m thinking of either handing out those client certificates or GPG. But those have problems as well. Not everyone (wants to) understand(s) GPG.

Other stuff

  • Did a lot of work on reducing the outstanding tickets in RT3 (mostly accounts, also some sysadmin stuff). I am worried about the number of tickets in the stalled state (waiting for a reply from someone other than sysadmin/accounts); I guess a lot of our answers go straight into the spam folder. Currently there are about 40 tickets are in that state (it was 20-25 before I started on the tickets). Can’t wait for the new mango possibilities.
  • Tried installing Zope and Plone. Gave up quickly. Seeing a security advisory on the Zope as well as the Plone site was also not encouraging. I’d rather have a rpm by Dag Wieers.

Library.gnome.org alpha 1

Goran Rakić started development on library.gnome.org as his Google Summer of Code 2006 project. For this he created a libgo script available in GNOME SVN in the library-web module.

It took a while to get it up and running, but the output of that script can now be seen at:

Help wanted
As you can see from the site, a lot is missing. As the last commit to library-web is over 7 months ago, I’m searching for anyone with Python, gtk-doc and gnome-doc-utils experience (or willing to learn) to help out. I’d appreciate people with an SVN account. Having volunteers with gnomeweb access would rock!

Note: I did not develop any of this code (or know much about it). As far as I know it was all made by Goran. Some of the documentation I found:

Action plan:
Please subscribe to gnome-web-list and be sure to read the archives (discussion took place over many, many months). I’d appreciate people who can work without much guidance, because I am pretty busy and do not know much about it.

Thanks
Above is not an attack or anything on Goran. I know he is busy and the existing code is very much appreciated. Just trying to get some people involved.

DSBL is not bad

Quoting from http://dsbl.org/faq:

  • list.dsbl.org single-stage relays tested by trusted testers
  • multihop.dsbl.org the outputs of multihop relays, tested by trusted testers
  • unconfirmed.dsbl.org everything else, including tests done by anonymous testers; people could potentially sign up their own ISP’s mail server to this list

After that it says the following:

Note that the multihop and unconfirmed lists are very aggressive and have the potential for a high level of false positives.

GNOME uses list.dsbl.org among others. Gmail is only listed by unconfirmed and multihop.

Dropping old Bug-Buddy reports

There is a proposal on desktop-devel-list to drop bug-buddy reports from either <=GNOME 2.14 or <=GNOME 2.16. I fully agree with <=GNOME 2.14 (receives max 5 bugreports/day). However, not sure about <=GNOME 2.16.

Based on server stats over the last week, we:

  • Created 23 GNOME <=2.14 bug-buddy reports. Update: To clarify, the <=2.14 bug-buddy programs updated themselves 9603 times (it does that when you start it and it wasn’t updated for at least 1 day, plus the server config changed). Obviously that method of reporting bugs is so broken that killing it wouldn’t have a big impact.
  • Received 5513 XML-RPC GNOME 2.15 + 2.16 bug-buddy reports. I assume 60% are auto-rejected (received!=created).
  • Received 475 XML-RPC bug-buddy report for other GNOME versions (sometimes the bug-buddy version couldn’t be determined).
  • Created 2405 bugreports in total (over the last 7 days instead of last week).

Seeing above stats, unless I get a good objections from many developers GNOME <=2.14 bug-buddy reports will be dropped (with an explanation message). Before dropping 2.15+2.16, I’d like some (developer) feedback on d-d-l. Please read the entire thread, then add your comments.

RHEL5
There is just one server left to upgrade – menubar. I’m delaying this after GNOME 2.18.1, plus I want to create a test installation first.

Upgrades continue

Servers that have been upgraded to RHEL5:

  • label
    Contains LDAP (GNOME usernames), live.gnome.org (it was down twice yesterday)

  • button
    Contains MySQL (reason for Bugzilla downtime yesterday), mango (GNOME account creation/update thingy)

  • window
    Hosts loads of websites, cvs.rpm.org, functions as master.gnome.org. It will be an ‘upgrade and see what breaks’.

  • box
    Hosts Bugzilla. In preparation I’ll move Bugzilla to window after it is upgraded RHEL5 (either today or tomorrow).

  • container
    Used for NFS. Upgraded while keeping most of the servers up (Bugzilla and some websites depend on NFS, but managed to work around that)

Servers to do and the current plan:

  • menubar – after more testing
    Known as mail.gnome.org. I do not want to upgrade it before having me/jdub do a test config on another server.

Benefits of RHEL5:

  • Our server configuration is documented
  • slapcat is very fast
  • Mango repository is now on svn.gnome.org (not publicly visible)
  • Account creation+foundation mails now use UTF-8
  • Learned how to change PHP4 DOM calls to PHP5
  • /usr/bin/php-cgi works so much better via CGI than /usr/bin/php
  • Stealing external interfaces is possible
  • Learned that the apparently unsupported RHEL upgrade option works
  • Learned that some sysadmins are crazy in their easy task consideration (ldap?!?)

It is very nice to work with lots of sysadmins on this.

Update: window and box have been branded RHEL5.