Slow ticket processing

I was questioned by Vincent at UDS if the GNOME sysadmin team needed help as some tickets took a while to process. We (me,Vincent) didn’t have the time to discuss it, but I mailed my thoughts to him. He suggested I’d blog about it. This is it. Below is the almost pristine mail that he received. He did suggest I’d make a summary, but it is late and I do not want to. I slightly reworded small parts though.
PS: These are my thoughts and plans. Haven’t discussed this on gnome-sysadmin.

There are a few problems:

  • Account requests take long to process
  • Mailing list requests take long to setup correctly

I am not aware of problems aside from above (meaning, AFAIK everything is running smoothly). Note that not everything
is sysadmin related. E.g. I am *very* slow with Bugzilla related bugs
(‘tickets’) currently due to lack of time. However, that is not a
sysadmin problem.

Long to process account requests
First of all, this is handled by the accounts team; not by the
sysadmin team. Although in practice there is an overlap. In short:
some sysadmins do not setup accounts; and some accounts people are not a sysadmin (have root). The delay is not just new accounts, also e.g.
updating the email address takes a long time.
I think this is basically caused by the following:

  • Some of the active accounts team have been scared away (some
    requests are so ‘friendly’ that a few accounts@ people quit).

  • Some people are just busy with other stuff
  • Handling accounts is a stupid boring task that should be automated for 99%; it is not fun
    nor rewarding (people have better things to do with their time)

Mailing lists setup takes a long time
I refuse to setup new mailing lists as it will only result in a broken
one. Nobody else seems to handle these within a reasonable timeframe,
so the end result is that they take months to process.

Solutions to the problems I stated initially that will not work:

  • Add people to the sysadmin team
  • Add people to the accounts team

Add more people to sysadmin team
From my experience I noticed that adding more people really
does not help (apart from making sure they can be trusted, are a
sysadmin, etc). The people who have the new privileges usually stop
helping after a few months at most. Although people do want to help,
it seems they don’t keep the interest.

Someone I would add is someone who continually says what needs to be
changed and how to do it (in a constructive way); plus knows what
should not be done. The reason for this is that as a sysadmin there is
not someone who will tell you either what to do or how to do it.
Meaning, a sysadmin will need to know that by heart.
In short: I want people who would join anyway, not people who want to help.

Add people to the accounts team
Due to reasons I specified before, I think it is better to automate
much of the task away (Summer of Code project) than spend lots of time to find more people to
do a boring task. Although I’d appreciate people helping out with
enhancing Mango.

Other causes:

  • Not every sysadmin follows RT3

Not every sysadmin follows RT3
Not sure why some do not have an account or they do not look at
tickets. They are subscribed to gnome-sysadmin though (which mostly receives cron/logwatch, etc etc).

I hope most problems can be solved by the following:

  • Implementation of new account creation system within mango

  • Redo the mailing list setup.
  • Avoid using RT3, make use of Bugzilla instead (sysadmin product).

New account creation system
This should provide transparency and automate most of the process.
This hopefully also includes ways to securely have people change their
account as well (e.g. add/remove SSH keys, update email address, etc).

Redo mailing list setup
Basically the mailing list setup is so strange that when a new mailing
list is created, it fails to work. Usually either the alias does not
exist (you cannot mail to it), the archive is not listed or the
archive does not work. I plan to redo the mailing list setup together
with the upgrade of that machine from RHEL3 to RHEL5.
Note: Currently it feels to me like a black box. Symlinks to symlinks,
etc. Also the available sysadmin instructions do not result in a
mailing list that actually works. This is also the reason the machine
still runs RHEL3 (I do not want to break it).
Oh, and the indexing of the archives is still broken.

Avoid using RT3
We currently require RT3 basically for the account system and generic
requests. This because we need a system which can mail random people
to setup accounts. Tickets are not shown as we have no way to verify
someone did not change the From:. The solution is sending people a
token and verifying that they mail this token back.
For the non-accounts uses I think we should be able to use Bugzilla in
theory. However, currently something that I do not want to explain publicly blocks a switch from RT3 to Bugzilla.
Note: gnome-sysadmin@gnome.org will stay for urgent requests and/or
when e.g. Bugzilla is down (which would be an urgent request).

3 thoughts on “Slow ticket processing”

  1. Great summary.

    Totally agree on the Bugzilla point, though I never thought I’d say that! But it’s so true — we need that in public.

    I’ve been figuring out the mailing lists stuff, working out how to fix it without losing archives or significantly breaking things… It runs really fast after the big fixes a few months back ;-) but the weird changes made with the archives due to disk space have made it extremely difficult to understand and fix. I desperately want to though, because I can’t clean up the mailing lists or fully migrate to lists.gnome.org without doing this first! Argh! ;-)

  2. I think sysadmins have been doing a good job technically, but a bad job building a “sysadmin community”.

    When I tried to join to sysadmin team, I was suscribed to the mailing list, I tried to some random stuff; but never I was able to do it, because my account wasn’t set up to do it. Every time I tried, a problem was arised.

    After 4 or 5 months I thought that my help wasn’t needed after all. It is natural to change the priorities; moreover if the job is done anyway.

    I’m not complaining against you or anyone. But if sysadmins ask for help, they should be able helping to be helped.

    Kind regards,

  3. Bugzilla still doesn’t have https support because of a) server load and b) how would you actually, in practice, attack bugzilla? There’s not much too attack…

    Nevertheless, this raises its ugly head if you want to put sysadmin requests in bugzilla – here’s an attack vector:

    a@b.com is a GNOME contributor
    nasty@c.com steals a@b.com‘s bugzilla password
    nasty@c.com requests some kind of important access on GNOME systems
    sysadmin approves it since a@b.com is a respected member of the community
    nasty@c.com owns you

    I am totally aware that this might be too paranoid, but it’s probably still worth bearing in mind.

Comments are closed.