Archive for the ‘computer’ Category

Wikimedia’s Road to Bugzilla 4.4

Thursday, March 6th, 2014

(How we puppetized, upgraded and moved Bugzilla to another server)

Though we currently also evaluate Wikimedia’s project management tools, we will have to stick with our current infrastructure for a while. Among many other tasks, I spent the last months preparing the upgrade of Wikimedia’s Bugzilla instance from 4.2 to 4.4. Some reasons for upgrading can be found in this Bugzilla comment.

In late November 2013 I started by cleaning up Wikimedia Bugzilla’s custom CSS which was copied about five years ago and not kept in sync. It turned out that 16 of 22 files could be removed as there was no sufficient difference to upstream’s default CSS code (Bugzilla falls back to loading the default CSS file from /skins/default if no custom CSS file is found in /skins/custom). Less noise and less diffing required for future upgrades. In theory.

After testing these CSS changes on a Wikimedia Labs instance and merging them into our 4.2 production instance, I created numerous patches and put them into Gerrit (Wikimedia’s code review tool) by diffing upstream 4.2 code, upstream 4.4 code, and our custom code.

At the same time, Wikimedia’s Technical Operations team wanted to move the Bugzilla server from the kaulen server in our old Tampa datacenter to the zirconium server in our new Ashburn (Eqiad) datacenter. While you’d normally prefer to do only one thing at a time, Daniel Zahn (of Technical Operations) and I decided to create a fresh Bugzilla 4.4 instance from scratch on the new server to see into which problems we would run. During this process Daniel Zahn turned the old setup on kaulen, which was largely manual and had organically grown over the years, into a proper Puppet module. For every “missing module” error we ran into we avoided installing anything from Perl’s CPAN in Bugzilla’s /lib folder and ensured we just rely on distribution packages, for a much cleaner install. Daniel Zahn installed the needed packages by adding them to puppet code. While doing this we also removed Bugzilla’s Sitemap extension as it created sporadic Search::Sitemap errors when running Bugzilla’s checksetup.pl (plus it’s unmaintained anyway). Furthermore I ran into another runtime error to fix.

After fixing all checksetup.pl issues and having Bugzilla accessible via a web browser, only Bugzilla’s upstream CSS was displayed instead of our custom CSS. Neither was Wikimedia’s custom CSS offered as an option in the browser, nor could I log into the new Bugzilla (to check which theme is set as default in the admin settings) as the database dump we used for testing predated the creation of my user account.

After Sean Pringle of Technical Operations deployed a more recent Bugzilla database dump I expected further problems due to upstream changes to CSS loading. I was happy to see that I had been wrong: No problems with our custom CSS theming anymore. Instead, I ran into problems with our custom “See Also” field changes: Adding and removing such URLs triggered errors, and URLs themselves were not displayed (but their corresponding “Remove” checkbox). Thanks to upstream help in #bugzilla on Mozilla IRC I finally found out that Perl’s use base instead of use parent was the culprit.

After creating symlinks to /extensions/WeeklyReport/ to avoid 404 errors for the “Weekly Bug Summary” link in the sidebar (our setup is slightly busted) and after fixing two problems with our cronjobs for whining and data collection we agreed on a date to copy the database, do some maintenance work, and switch the DNS entry. This was announced one week in advance by adding a banner to Bugzilla via its announcehtml parameter.

A few hours before the switch on February 12th 2014, Daniel lowered the Time-to-live (TTL) values of the DNS entry of our Bugzilla. When the migration started, I set Bugzilla’s shutdown parameter to make the web UI inaccessible and also the WebService API return a 503 error for the Bingle script that syncs Bugzilla with Wikimedia’s Mingle instance. This was important to make sure that nobody can write anymore to the old database. We updated the IRC channel topic in #wikimedia-tech to tell that Bugzilla is under scheduled maintenance and logged the action in #wikimedia-operations so it got added to Wikimedia’s Server admin log. All in all we had only forgotten two minor things: Our Gerrit integration (a bot adding Gerrit notifications about related patches as comments in Bugzilla) bot was not able to write and got a 503 error back – Chad quickly disabled it. And our Nimsoft watchmouse sent an “ALERT! Bugzilla: Service Temporarily Unavailable” message to the Operations mailing list.

Sean Pringle migrated the old database from db9 in Tampa, to a new database on db1001 in Eqiad. After this was done, Daniel Zahn ran checksetup.pl to apply the scheme upgrades needed for 4.4.

After 30 minutes of testing to make sure everything works as expected we deployed two more custom patches: Showing common queries on the frontpage and making saved reports work. While having the downtime I also switched off bugmail to do some mass-changes without spamming everybody: I merged some version numbers in the “MediaWiki” product to have a shorter Version dropdown, removed the wikibugs-l watcher account from some bug reports as it is unneeded (set as a global watcher in Bugzilla anyway, hence a potential issue if a ticket was moved to a restricted product like “Security” still triggering public bugmail).

A few minutes before the end of the announced downtime of three hours, Daniel switched DNS so the new Bugzilla on the new server became available to the public. A few hours later, to work around isses for clients not supporting SNI, Daniel changed the order in which Apache loads virtual hosts. This ensures that older clients like Internet Explorer on Microsoft Windows XP will always get to see Bugzilla instead of other miscellaneous web services sharing the same hardware. I had also overlooked a small UI issue that I fixed two days later.

Now that all is done, the result can now be seen on bugzilla.wikimedia.org. All steps to upgrade Wikimedia Bugzilla from 4.2 to 4.4 were documented on a wiki page. You can find all of our custom modifications here.

Lack of Maintainership & Finding a Project to Contribute to

Thursday, January 30th, 2014

One of the things I liked about the GNOME Documentation Hackfest (apart from the hospitality of Kat, Dave and the University of East Anglia) was the opportunity of teachers and students popping in and discussing open source project management related aspects with us.

One obvious topic is contributor involvement.
The Number Two question of people interested in contributing to GNOME (right after “How do I get started?”) is “I know programming language XYZ. Which projects would be good?”. Which leads to the question:
What are obstacles for contributors to find projects in language XYZ themselves?

And as volunteer based projects allow project maintainers or translation team leaders to disappear without a warning, new contributors might not realize or know where to escalate in order to become a new maintainer, especially if the single contact point of failure is a private email address of the previous maintainer who will never reply. New contributors do not know how to find out how “maintained” a project is – they would have to find the “log” page of the project on GNOME Git plus understand that translation updates are not a sign of development activity.
How to help contributors avoid writing a patch for an unmaintained dead rotten project where nobody will ever notice and appreciate their contribution?

This problem actually affects a larger group: Translators, documentators and bug reporters all waste time working on projects that will never see a release again. One year ago, one third of the non-archived modules in GNOME Git had not seen any code activity for more than two years. Obviously, these dead projects are not a good idea to contribute to if you have no experience in software project management yet but just want to contribute a small patch that scratches your own itch.

And as we discussed this I pointed to a university paper I wrote a year ago, comparing Apache and GNOME.
Realizing how often I refer to it it’s probably useful to link the PDF file:

Indicators for Missing Maintainership in Collaborative Open Source Projects (PDF)

Very quickly after that discussion, Frédéric created a “GNOME Project Health” webpage based on metrics in that paper.
Frédéric’s page allows to see which listed GNOME maintainers are active, how active the project is (the higher the score, the less activity exists), and in which programming language(s) the project is written. It’s a great start and I owe Frédéric a beer for it.

If you are a GNOME project maintainer you can help improving it:
If your project is mostly written in the programming languages X and Y, add the lines

<programming-language>X</programming-language>
<programming-language>Y</programming-language>

to the .doap file in the top level folder in Git, and check if the maintainers listed in the DOAP file do not collide with the reality out there.

GNOME Documentation Hackfest, Day 0

Sunday, January 26th, 2014

Reporting live from GNOME Docs Hackfest at Norridge, East Anglia.

Fréd arrived after bypassing some tree falling on his way. But he was very late. Shaun is still in a snowstorm. Ryan has escaped that already. But fearless leader Kat, Dave, Julita, Phil, Michael, and Baptiste made it. Petr plans to arrive in the middle of the night. If his plane does not decide to make even more problems.

Julita plans to refresh the structure of the Evolution user docs to make it more compact plus might take a peek at GNOME Developer Platform Demos.

Phil “arrived on a train, that was pretty good, and I’ve actually resolved a third of the licensing issues of GNOME system monitor.”

Michael and Baptiste visited the Protestant cathedral. Michael worked on system monitor and taking screenshots. Baptiste also kept contact to Fréd so he does not feel too alone and lost. And Sindhu joined remotely.

I went through numerous documentation related bug reports and updated them (plus as a side effect created statistics about all those many unmaintained modules in GNOME Git).
For a short moment I also wondered: Why does GNOME push for using AppData files if we have extensible DOAP files already in each repository? Hmm.

Interested documentation fans in the backyard.

Documentation fans in the backyard of the venue (photo by Michael Hill)

Wikimedia in Google Code-In 2013

Wednesday, January 22nd, 2014

Google Code-In 2013 Logo

Google Code-In (GCI) 2013 is over and the winners have been announced! Congratulations to everybody!

Quim Gil and I organized the participation of Wikimedia in GCI this year. We set up a central wikipage that we pointed students to for recurring general questions. Mostly these were about expectations on communication, how to use IRC, how to set up a development environment and the toolchain (Git, Gerrit, Bugzilla), plus a list of our mentors and their IRC nicknames and timezones (some enthusiastic students welcome being reminded of timezones, plus having patience is one of the good lessons to learn).
We also had a section for mentors explaining how to write good GCI tasks and which sentences and information should be part of every task description.

For the Wikimedia Engineering Community Team, GCI was a helpful lesson to prepare improving our Annoying little bugs landing page for new contributors (still on my TODO list). Students hopefully found interest in contributing to a great FOSS community (I have seen numerous students who continue contributing and are still active after the contest has finished). I hope that Wikimedia will be able to invite the most active students to the Wikimania conference.

We ended up with way more tasks available than expected (and more than 200 successfully finished), had a number of students who really impressed us by code quality (number of patch reviews required) and speed, and Quim managed surprisingly well to convince established developers to become mentors. Also, the nine hours of time difference between Quim and me was a big advantage in order to respond quickly to requests.

All together, GCI was successful and participating in GCI was good decision, contrary to my initial reluctance.
I would like to thank Google and all involved mentors and students for their hard work and for making this possible.

Manually Installing Bugzilla 4.4 on Fedora 20

Saturday, January 18th, 2014

One rainy winter I tried to manually install Bugzilla 4.4 on a Fedora 20 machine by using Bugzilla’s latest stable upstream code instead of the Bugzilla package provided by Fedora. You, being one of my delighted long-term blog readers, already know that one of my favorite hobbies is to run into issues. Generally. Hence allow me to share them with you here, to some extend.

General hint to the reader: RTFM: installation guide. Also ignore /etc/bugzilla/localconfig as it is only used by the Bugzilla package provided by Fedora.

  • cd /usr/share/
  • Download Bugzilla 4.4 via Bazaar (which will get replaced by Git soon): bzr co bzr://bzr.mozilla.org/bugzilla/4.4 bugzilla44
  • cd /usr/share/bugzilla44/
  • Run checksetup.pl to see that I should install numerous packages for functionality. In my case this meant yum install perl-Template-Toolkit perl-Template-GD perl-GDGraph perl-GDGraph3d perl-GDTextUtil perl-MIME-tools perl-HTML-Scrubber perl-TheSchwartz (you can find a nice list of package names on Fedora here).
  • Create a symlink so your Apache web server will find your Bugzilla: ln -s /usr/share/bugzilla44/ /var/www/html/bugzilla44
  • ./checksetup.pl --check-modules – if all packages are sorted out it should now complain about the database.
  • Start the web server and database services: systemctl start httpd.service and systemctl start mysqld.service
  • Database setup
    • Reset the MariaDB/MySQL password in case you forgot: mysqladmin -u root password
    • mysql -u root -p (then enter the password)
    • mysql$> CREATE DATABASE bugs;
    • mysql$> GRANT ALL PRIVILEGES ON bugs.* TO andre@localhost IDENTIFIED BY ‘password123′;
    • Edit /usr/share/bugzilla44/localconfig and insert the previously defined database information:
      db_host = 'localhost';
      db_user = 'bugs';
      db_user = 'andre';
      db_pass = 'password123';
  • Giving /usr/share/bugzilla44/testserver.pl http://localhost/bugzilla44 a try cannot hurt either to find further problems.

Problems

Now let’s finally start running into problems with Apache and SELinux (manuals FTW?).

  • Problem: CGI content displayed as plain raw text in the browser:
    • Remove the preceding # of the line #AddHandler cgi-script .cgi in /etc/httpd/conf/httpd.conf
  • Problem: 403 Forbidden error
    • Check that $webservergroup in /usr/share/bugzilla44/localconfig is set to apache
    • chown -R 751 /usr/share/bugzilla44
    • chown root:apache -R /usr/share/bugzilla44
    • Check /var/log/httpd/error_log and see the line
    • Options ExecCGI is off in this directory: /var/www/html/bugzilla44/index.cgi. So edit /etc/httpd/conf/httpd.conf and add:
      <Directory "/var/www/html/bugzilla44">
      AddHandler cgi-script .cgi
      Options +ExecCGI +FollowSymLinks
      DirectoryIndex index.cgi
      AllowOverride Limit FileInfo Indexes Options
      Order allow,deny
      Allow from all
      Require all granted
      </Directory>

      (I run Bugzilla on a test machine and don’t need to care about a secure setup, so my settings might not be good settings for you, obviously.)

  • Problem: 500 Internal Server Error
    • Check /var/log/httpd/error_log and see the line /var/www/html/bugzilla44/.htaccess: Options not allowed here
    • Instead of commenting out lines, double-check that $create_htaccess = 0; is set in /usr/share/bugzilla44/localconfig and delete /usr/share/bugzilla44/.htaccess before it complains about every damn single line in there. Run checksetup.pl, as usual.
  • Problem: Try to set up any parameter on the Bugzilla admin page and fail
    • Software error: Error in tempfile() using template data/params.XXXXX: Could not create temp file data/params.abcDE: Permission denied at Bugzilla/Config.pm line 270.
    • Wait, obscure and cryptic permission issues even after running chmod a+rwx /usr/share/bugzilla44/data/? That sounds like the main area of expertise of SELinux! But random stuff I found on the interwebs like setsebool -P httpd_enable_homedirs on and setsebool -P httpd_read_user_content on did not help. echo 0 >/selinux/enforce neither (that command likely changed over the last years, see next sentence).
    • Following a blogpost about Nagios vs SELinux, running setenforce 0 to change SELinux’ mode from “enforcing” to “permissive” (you can check via sestatus) finally makes Bugzilla work as expected. Obviously looking at the SELinux context of a specific file via ls -Z and adjusting it via chcon is less pervasive, but I don’t care much about SELinux.

Bugzilla now works for me.

(General hint for people from the future who found this blogpost after searching for how to fix their Bugzilla problem: If you need Bugzilla support, send an email to Bugzilla support instead of using the comment function on somebody’s blog. Cheers.)

2014: Plans.

Monday, January 6th, 2014

Studying bug management

To my disappointment my university seems to have no expertise available to mentor a diploma thesis about best practices and common problems in open source bug management. After ten years of working in this field there are quite some open questions I’d like to see answered and after reading lots of scientific papers and books I yet have to find one covering the many aspects and personas involved, so why not write it?
I plan to do some case study research across FLOSS projects (and I’m thankful to Mr Riehle of the University of Erlangen-Nuernberg who pointed out a good resource explaining how to properly do this).

If you are by any chance a teacher at a German university and could imagine to mentor such a thesis, please feel free to contact me.

Wikimedia Bugzilla

This is worth a separate blog post soon, so for the time being I just point to my task list and status updates for the latest achievements. I also plan to attend FOSDEM which has a dedicated wiki track this year!

GNOME

There is a documentation hackfest at the end of this month and I hope to work on the Evolution user docs which I maintain, followed by attending FOSDEM in Brussels and DevConf in Brno.

Chennai

I plan to spend quite some time in Chennai this year by visiting several times. Drop me a line if you live there and would like to discuss Wikimedia, GNOME, or bug management!

Mozilla Summit 2013

Tuesday, October 8th, 2013

mozsummit2013

I had the pleasure to attend the European edition of Mozilla Summit 2013 last weekend. It also took place in Santa Clara (USA) and Toronto (Canada), hosting nearly 2000 Mozillians in total.

The session which interested me the most was discussing bugzilla.mozilla.org (bmo) and the relation to upstream Bugzilla development (especially getting more great stuff from bmo into upstream so other Bugzilla instances could use it). I try to follow bmo development (glob’s “push day” posts and the planning page are the most useful resources) and especially the recent bmo experiments with product dashboards, user dashboards and user pages, but getting an overview of all plans “in a nutshell” is always helpful, especially as there aren’t many upstream meetings.

FYI, further plans for the bmo instance include

  • Native REST API
  • moving the code repository from bzr to git
  • UI improvements, more AJAX
  • upstreaming the new bmo skin
  • moving flags to its own database table for performance reasons (flags are heavily used for release management and hence frequently added in bmo)
  • Investigating the use of memcached
  • Integration with ReviewBoard to replace Splinter

Having spent a few years with Maemo and MeeGo, I also loved the discussion and analysis of the mobile market and its players (competitors and potential allies of Firefox OS).

All in all the conference was extremely well organized and very welcoming – lots of smiling faces. I enjoyed discussing best bug management practices with other triagers, but also offtopic stuff like middle east politics or the times of surveillance that we’re living in. And my hotel roommate awoke my interest in profiling the JavaScript of my triage helper tools so they now run way faster.

Know more, do more, do better.

I would like to express my gratitude to Mozilla for inviting me to this event.

Shades of Blue

Wednesday, October 2nd, 2013

I’m not entirely sure what’s special about my machine (T61 Thinkpad) that makes it lock up and reboot at least once per day without a warning (but sometimes after showing things in blue), but I’ve decided to blame something down the Xorg/Nouveau stack.
Interestingly, SSH’ing onto the sick machine isn’t much help either – when it freezes, cat /proc/kmsg, tail -f /var/log/Xorg.0.log and /var/log/messages, or gdb attached to /usr/bin/Xorg and gnome-shell don’t react anymore and don’t show any “additional” output either. Maybe it’s time to try Wayland? Can’t get much worse. :P

gnome-shell, now with more blue!

Bugzilla Tips (XI): Reports: Tickets closed last week by resolution

Friday, August 30th, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

This episode goes deeper into Creating reports and tables and Bugzilla’s Advanced Search.

Three weeks ago a development team asked how to get the number of Bugzilla tickets closed in the last seven days by some resolutions (e.g. FIXED, WONTFIX, INVALID, WORKSFORME) while not being interested in the number of tickets closed as DUPLICATE or LATER.

Another usecase for tabular reports in Bugzilla: The vertical axis will display the components which the team maintains and the horizontal axis will display the bug report resolutions:

bugzilla-closedstats1

We manually selected the components of the team in the “Components” list (keep the Control key pressed to make a multi-selection), and selected all resolutions which should be listed in the bug report (see here for the meaning of the “—” resolution). Keep in mind that these resolutions only refer to the current status of the bug reports.

bugzilla-closedstats2

As we only want to know the number of tickets who were resolved in the last week, we need to query for changes of the resolution. This is where the Custom Search comes into play which allows logical combinations of conditions. For our example:

bugzilla-closedstats3

The generated report looks like this:

bugzilla-closedstats4

Though we chose to include INVALID and WONTFIX resolutions in the search criteria above, the table does not have INVALID and WONTFIX columns because no tickets were closed with that resolution in the last week. Similarly there is no row for the “CLDR” component chosen in the search criteria above because no CLDR ticket was resolved in the last week.

Above example explains how to exclude statuses from the table which you are not interested in. If you want all resolutions displayed in the table anyway (and as setting a resolution requires changing the bug status to RESOLVED), you might already guess that there is an easier way by using “Search By Change History”:

bugzilla-closedstats5

Now in case you wonder why setting the resolutions via the custom search is not sufficient and why it is also required to set the current resolution in the “Product / Component / Status / Resolution” lists above as search criteria: It avoids including tickets which already got reopened in the meantime.

If the numbers are slightly different than your expectations, note that until Bugzilla version 4.2, -7d refers to the beginning of the day in the timezone that Bugzilla is set to, not to the last 168 hours. This has changed in Bugzilla 4.4 though, -1d refers to the last -24h while -1ds refers to the start of the day.

Bugzilla Tips (X): Triage helper tools: Greasemonkey scripts

Friday, August 23rd, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

If you read a lot of Bugzilla tickets per day you run into some recurring situations. For example, a bug report might miss sufficient information and you want to point its reporter to a wikipage which explains how to write good bug reports, or you clean up older rotting tickets without enough information and while closing them you want to explain to the reporter why you close the report.

In such situations, having some one-click stock answers (or “canned responses” as others call them) can come in handy in order to save time. I’ve been using several Greasemonkey scripts in my Firefox browser over those years. The two Javascript files which I use in Wikimedia Bugzilla are available to everybody in the code repository. They can be checked out by running the command git clone https://gerrit.wikimedia.org/r/p/wikimedia/bugzilla/triagescripts.git
To use them in Firefox, one has to install Greasemonkey, visit the Git web interface of the repository with Firefox and click the “Raw” links. An installation dialog will open. Note that these scripts only work if you have canconfirm and editbugs permissions in Bugzilla.

This is how it looks after installing the scripts:

bugzilla-triagehelpers-stockanswers

In the picture above, I clicked the “[Close:WorksForMe]” answer. An explanatory command (which also automatically picks up the name of the reporter) is added and the status “RESOLVED WORKSFORME” is set.

As time goes by I’ve added more functionality for my convenience, mostly links to places which are related to functionality exposed in the Bugzilla interface:

bugzilla-triagehelpers

Most useful to me is

  • coloring reporters based on their email addresses (I might spend less attention on a report created by an established developer or employee than a newcomer),
  • coloring the component (MediaWiki has hundreds of extensions, and extensions which are deployed on Wikimedia servers might receive more attention than a non-deployed 3rd party extension),
  • a link to search for other reports created by the same user, and
  • a link to a graph of the priority distribution of tickets for the component (to check how realistically priorities are set – if you only have open tickets with high priority for your component, then something is wrong).

While the code of these Greasemonkey scripts would welcome some cleanup and refactoring, it works for me. Plus today I finally introduced a bunch of boolean variables at the top of the scripts so users can easily define which functionality s/he wants to enable (or not). You are welcome to give it a try (and provide patches if you feel like hacking away).

Of course all this functionality could also be added to the Bugzilla code on the server but I do not want to clutter the Bugzilla user interface even more for everybody by default.
Note that there is also a “proper” upstream Bugzilla extension called Canned Comments available since May 2012 which I have not played with yet.