Slow ticket processing

I was questioned by Vincent at UDS if the GNOME sysadmin team needed help as some tickets took a while to process. We (me,Vincent) didn’t have the time to discuss it, but I mailed my thoughts to him. He suggested I’d blog about it. This is it. Below is the almost pristine mail that he received. He did suggest I’d make a summary, but it is late and I do not want to. I slightly reworded small parts though.
PS: These are my thoughts and plans. Haven’t discussed this on gnome-sysadmin.

There are a few problems:

  • Account requests take long to process
  • Mailing list requests take long to setup correctly

I am not aware of problems aside from above (meaning, AFAIK everything is running smoothly). Note that not everything
is sysadmin related. E.g. I am *very* slow with Bugzilla related bugs
(‘tickets’) currently due to lack of time. However, that is not a
sysadmin problem.

Long to process account requests
First of all, this is handled by the accounts team; not by the
sysadmin team. Although in practice there is an overlap. In short:
some sysadmins do not setup accounts; and some accounts people are not a sysadmin (have root). The delay is not just new accounts, also e.g.
updating the email address takes a long time.
I think this is basically caused by the following:

  • Some of the active accounts team have been scared away (some
    requests are so ‘friendly’ that a few accounts@ people quit).

  • Some people are just busy with other stuff
  • Handling accounts is a stupid boring task that should be automated for 99%; it is not fun
    nor rewarding (people have better things to do with their time)

Mailing lists setup takes a long time
I refuse to setup new mailing lists as it will only result in a broken
one. Nobody else seems to handle these within a reasonable timeframe,
so the end result is that they take months to process.

Solutions to the problems I stated initially that will not work:

  • Add people to the sysadmin team
  • Add people to the accounts team

Add more people to sysadmin team
From my experience I noticed that adding more people really
does not help (apart from making sure they can be trusted, are a
sysadmin, etc). The people who have the new privileges usually stop
helping after a few months at most. Although people do want to help,
it seems they don’t keep the interest.

Someone I would add is someone who continually says what needs to be
changed and how to do it (in a constructive way); plus knows what
should not be done. The reason for this is that as a sysadmin there is
not someone who will tell you either what to do or how to do it.
Meaning, a sysadmin will need to know that by heart.
In short: I want people who would join anyway, not people who want to help.

Add people to the accounts team
Due to reasons I specified before, I think it is better to automate
much of the task away (Summer of Code project) than spend lots of time to find more people to
do a boring task. Although I’d appreciate people helping out with
enhancing Mango.

Other causes:

  • Not every sysadmin follows RT3

Not every sysadmin follows RT3
Not sure why some do not have an account or they do not look at
tickets. They are subscribed to gnome-sysadmin though (which mostly receives cron/logwatch, etc etc).

I hope most problems can be solved by the following:

  • Implementation of new account creation system within mango
  • Redo the mailing list setup.
  • Avoid using RT3, make use of Bugzilla instead (sysadmin product).

New account creation system
This should provide transparency and automate most of the process.
This hopefully also includes ways to securely have people change their
account as well (e.g. add/remove SSH keys, update email address, etc).

Redo mailing list setup
Basically the mailing list setup is so strange that when a new mailing
list is created, it fails to work. Usually either the alias does not
exist (you cannot mail to it), the archive is not listed or the
archive does not work. I plan to redo the mailing list setup together
with the upgrade of that machine from RHEL3 to RHEL5.
Note: Currently it feels to me like a black box. Symlinks to symlinks,
etc. Also the available sysadmin instructions do not result in a
mailing list that actually works. This is also the reason the machine
still runs RHEL3 (I do not want to break it).
Oh, and the indexing of the archives is still broken.

Avoid using RT3
We currently require RT3 basically for the account system and generic
requests. This because we need a system which can mail random people
to setup accounts. Tickets are not shown as we have no way to verify
someone did not change the From:. The solution is sending people a
token and verifying that they mail this token back.
For the non-accounts uses I think we should be able to use Bugzilla in
theory. However, currently something that I do not want to explain publicly blocks a switch from RT3 to Bugzilla.
Note: gnome-sysadmin@gnome.org will stay for urgent requests and/or
when e.g. Bugzilla is down (which would be an urgent request).

Bugzilla 3.0

Part of this was copy/pasted from the announcement.

Here’s just a sampling of the major new features in version 3.0:

  • Less ugly perl code (almost everything is readable&understandable now)
  • Custom Fields
  • mod_perl support for greatly-improved performance
    While sucking up loads of memory 🙁

  • Per-Product Permissions
    non hacky version of what is available on GNOME Bugzilla (patch wasn’t by me)

  • XML-RPC Interface; see the WebService stuff on this page
  • Create and Modify Bugs by Email
    Still not GPG authenticated though; so I won’t enable it on b.g.o without at least a hack to make it more secure.

  • And even more. See all the new features at: http://www.bugzilla.org/releases/3.0/new-features.html

GNOME Bugzilla
I am really busy. Help with the port is appreciated. The current (very rough and unfinished) port is available in the bugzilla.gnome.org SVN module; while the site currently uses the bugzilla-newer SVN module. I think that probably the port should use Bugzilla HEAD (what will eventually be 3.2) though. I basically need at least 3.0 to easily avoid those bug-buddy crash duplicates.

Name for the build-brigade machine

In the near future I’m going to setup the build-brigade machine. This machine is currently known as tinderbox1. As we’ll be using buildbot, that machine name is wrong (the ‘1’ also isn’t nice). I could just name it buildbot, but ideally a more generic name would be nice.

Suggestions?

Please add comments to the blog or as a reply to this mail (just mail build-brigade-list@gnome.org)