Archive for the ‘wikimedia’ Category

Good bye Bugzilla, welcome Phabricator.

Wednesday, December 17th, 2014

<tl;dr>: Wikimedia migrated its bug tracking from Bugzilla to Phabricator in late November 2014.

After ten years of using Bugzilla with 73681 tickets and ~20000 user accounts and after months of planning, writing migration code, testing, gathering feedback, discussing, writing more code, writing documentation, communicating, et cetera, Wikimedia switched from Bugzilla to Phabricator as its issue tracking tool.
Phabricator is a fun adventure game collaboration platform and a forge that consists of several well-integrated applications. Maniphest is the name of the application for handling bug reports and tasks.
My announcement from May 2014 explained the idea (better collaboration and having less tools) and the decision making process that led to choosing Phabricator and starting to work on making it happen.

Wikimedia Phabricator frontpage an hour after opening it for the public again after the migration from Bugzilla.

Wikimedia Phabricator frontpage an hour after opening it for the public again after the migration from Bugzilla.

Quim already published an excellent summary of Wikimedia Phabricator right after the migration from Bugzilla, covering its main features and custom functionality that we implemented for our needs. Read that if you want to get an overview of how Phabricator helps Wikimedia with collaborating and planning in software development.
This blog post instead covers more details of the actual steps taken in the last months and the migration from Bugzilla itself. If you want even more verbose steps and information on the progress, check the status updates that I published every other week with links to the specific tickets and/or commits.

Preparation

After reviewing our project management tools and closing the RfC the team started to implement a Wikimedia SUL authentication provider (via OAuth) so no separate account is needed, work on an implementation to restrict access to certain tasks (access restrictions are on a task level and not on a project level), and creating an initial Phabricator module in Puppet.
We started to discuss how to convert information in Bugzilla (keywords, products and components, target milestones, versions, custom fields, …), which information to entirely drop (e.g. the severity field, the history of field value changes, votes, …), and which information to only drop as text in the initial description instead of a dedicated field. More information about data migrated is available in a table. This constantly influenced the scope of the script for the actual data migration from Bugzilla (more information on code).

We already had a (now defunct) Phabricator test instance in Wikimedia Labs under fab.wmflabs.org which we now started to also use for planning the actual migration.
There’s a 7 minute video summary from June describing the general problem with our tools that we were trying to solve and the plan at that time. We also started to write help documentation.

As we got closer to launching the final production instance on phabricator.wikimedia.org, we decided to split our planning into three separate projects to have a better overview: Day 1 of a Phabricator Production instance in use, Bugzilla migration, and RT migration.

On September 15th, phabricator.wikimedia.org launched with relevant content imported from the fab.wmflabs.org test instance which we had used for dogfooding. In the case of Wikimedia, this required setting up SNI and making it work with nginx and the certificate to allow using SUL and LDAP for login. After the production instance had launched we also had another Hangout video session to teach the very basics of Phabricator.

To provide a short impression of further stuff happening in the background: Elasticsearch was set up as Phabricator’s search backend, some legal aspects (footer, determining the license of submitted content) were brought up, phab-01.wmflabs.org was set up as a new playground, and we made several further customizations when it comes to user-visible strings and information on user pages within Phabricator. In the larger environment of Wikimedia infrastructure interacting with the issue tracker, areas like IRC bots, interwiki links, on-wiki templates, and automatic notifications in tasks about related patches in the code review system were dealed with or being worked on.

Paying attention to details: The “tracked” template on Wikimedia sites supports linking to tasks in Phabricator, while still redirecting links to Bugzilla tickets via URL redirects (see below).

We also had a chicken and egg problem to solve: Accounts versus tickets. Accounts in Bugzilla are defined by email addresses while accounts in Phabricator are user names. For weeks we were asking Bugzilla users and community users to already create an account in Phabricator and “claim” their Bugzilla accounts by entering the email address that they used in Bugzilla in their Phabricator account settings. The plan was to import the tickets and account ‘placeholders’ and then use cron jobs to connect the placeholder accounts with the actual users and to ‘claim’/connect their past Bugzilla contributions and activity by updating the imported data in Phabricator.

On October 23th, we made a separate “bugzillapreview” test instance available on Wikimedia Labs with thousands of Bugzilla tickets imported. For two weeks, the community was invited to check how Bugzilla tickets are going to look in Phabricator after the migration and to identify more potential issues. The input was helpful and valuable: We received 45 reports and fixed 25 of them (9 were duplicates, 2 invalid, and 9 got declined).

A task imported from Bugzilla in the Phabricator preview instance.

A task imported from Bugzilla in the Phabricator preview instance.

Having had reached a good overview, we created a consolidated list of known issues and potential regressions created by the migration from Bugzilla to Phabricator and defined a final date for the migration: November 21-23.

Keeping timestamps of comments intact (such as the original creation date of a ticket in Bugzilla or when a certain comment was made) was still something to sort out at this point (and got tackled). It would have been confusing and would have broken searches that triagers need when trying to clean up (e.g. tickets which have not seen updates for two years).

It was also tricky performance-wise to keep the linear numbering order of reports which was requested by many people to not solely depend on URL redirects from bugzilla.wikimedia.org to phabricator.wikimedia.org which we planned to set up (more information on the redirect setup). As we already had ~1400 tickets in Phabricator we went for the simple rule “report ID in Bugzilla + 2000 = task ID in Phabricator”.

Regarding documentation and communication, we created initial project creation guidelines, sent one email to those 66 users of personal tags in Bugzilla warning that tags will not be migrated, sent two emails to the 850 most recently active Bugzilla users asking them to log into Phabricator and provide their email address used in Bugzilla to claim their imported contributions as part of the migration already (for comparison, the average number of active users per month in Bugzilla was around 500+ for the last months), put migration announcement banners on mediawiki.org and every page on our Bugzilla, sent reminders to the wikitech-l, mediawiki-l, wikitech-ambassadors, and wmfall mailing lists.

After a last ‘Go versus No-Go’ meeting on November 12th, we set up the timeline with the list of steps to perform for the migration, ordered, with assignees defined for each task. This was mostly based on the remaining open dependencies of our planning task. We had two more IRC office hours on November 18 and 19 to answer questions regarding the migration and Phabricator itself.

While migrating, the team used a special X-Forwarded-For header to still be able to access Bugzilla and Phabricator via their browsers while normal users trying to access Phabricator or Bugzilla were redirected to a wikipage telling them what’s going on and where to escalate urgent issues (MediaWiki support desk or IRC) while no issue tracker is available. With aforementioned URL redirects in place we intended to move and keep Bugzilla available for a while under the new address old-bugzilla.wikimedia.org.

Migration

The page on mediawiki.org that users were redirected to while the migration was taking place.

The page on mediawiki.org that users were redirected to while the migration was taking place.

The migration started by switching Bugzilla to read-only for good. Users can still log into Bugzilla (now available at old-bugzilla.wikimedia.org) and e.g. run their searched queries or access their list of votes on the outdated data but they cannot create or change any existing tickets.

We pulled phabricator.wikimedia.org and disabled its email interface, switched off the code review notification bot for Bugzilla, and switched off the scripts to sync Bugzilla tickets with Mingle and Trello.

The data migration started by applying a hack to workaround a Bugzilla XML-RPC API issue (see below), running the migration fetch script (tasks and comments), reverting the hack, running the migration create script (attachments), moving Bugzilla to old-bugzilla.wikimedia.org, starting the cron jobs to start assigning Bugzilla activity to Phabricator users by replacing the generic “bzimport” user by the actual corresponding users, and setting up redirects from bugzilla.wikimedia.org URLs.

A task before and after users have claimed their previous Bugzilla accounts (positions of comments in the right image manually altered for better comparison).

A task before and after users have claimed their previous Bugzilla accounts (positions of comments in the right image manually altered for better comparison).

After several of those data migration steps we performed numerous tests. In parallel we prepared emails and announcements to send out and publish once we’re finished, updated links to Bugzilla by Phabricator on dozens of wikipages, updating MediaWiki templates on the Wikimedia, and further small tasks.

Paying attention to details: The "infobox" template on MediaWiki extension homepages linking to the extension's bug reports at the bottom, now handled in Phabricator instead of Bugzilla.

Paying attention to details: The infobox template on MediaWiki extension homepages linking to the extension’s bug reports at the bottom, now handled in Phabricator instead of Bugzilla.

For those being curious about time spans: Fetching the 73681 Bugzilla tickets took ~5 hours, importing them ~25 hours, and claiming the imported user contributions of the single most active Bugzilla user took ~15 minutes.

But obviously we were pioneers that could not rely on Stackoverflow.
Even if you try to test everything, unexpected things happen while you are running the migration. I’m proud to say that we (well, rather Chase, Daniel, Mukunda and Sean when it came to dealing with code) managed to fix all of them. And while you try to plan everything, for such a complex move that nobody has tried before, there are things that you simply forget or have not thought about:

  • We had to work around an unresolved upstream XML-RPC API bug in Bugzilla by applying a custom hack when exporting comments in a first step and removing the hack when exporting attachments (with binary data) in a second step. Though we did, it took us a while to realize that Bugzilla attachments imported into Phabricator were scrambled as the hack got still applied for unknown reasons (some caching?). Rebooting the Bugzilla server fixed the problem but we had to start from scratch with importing attachments.
  • Though we had planned to move Bugzilla from bugzilla.wikimedia.org to old-bugzilla.wikimedia.org after exporting all data, we hadn’t realized that we would need a certificate for that new subdomain. For a short time we had an ugly “This website might be insecure” browser warning when users tried to access the old Bugzilla until old Bugzilla was moved behind the Varnish/nginx layer with its wildcard *.wikimedia.org certificate.
  • Two Bugzilla statuses did not get converted into Phabricator tags. The code once worked when testing but broke again later at some point without anybody realizing but this was noticed and fixed.
  • Bugzilla comments marked as private got public again once the cron jobs claiming contributions of that commenter were run. Again this was noticed and fixed.
  • We ended up with a huge feed queue and search indexing queue. We killed the feed daemon at some point. Realizing that it would have taken Phabricator’s daemons ~10 days to handle the queue, Chase and Mukunda debugged the problem together with upstream’s Evan and found a way to improve the SQL performance drastically.
  • We hadn’t thought about switching off some Bugzilla related cronjobs (minor) and I hadn’t switched off mail notifications from Bugzilla so some users still received "whining" emails until we stopped that.
  • We had a race condition in the migration code which did not always set the assignee of a Bugzilla ticket also as the assignee of the corresponding task in Phabricator. We realized early enough by comparing the numbers of assigned tickets for specific users and fixed the problem.
  • I hadn’t tested that aliases of Bugzilla reports actually get migrated. As this only affected ~120 tickets we decided to not try to fix this retroactively.
Phabricator daemons being (too) busy handling the tasks mass-imported from Bugzilla.

Phabricator daemons being (too) busy handling the tasks mass-imported from Bugzilla.

We silently reopened Phabricator on late Sunday evening (UTC) and announced its availability on Monday morning (UTC) to the wikitech-l community and via the aforementioned blogpost.

A list of dependency tasks handled before completing the migration from Bugzilla to Phabricator is available.

Advantages

Phabricator has many advantages compared to Bugzilla: Wikimedia users do not reveal their email addresses and users do not have another separate login and password. (These were the most popular complaints about Bugzilla.)

Integration with MediaWiki's Single User Login via OAuth - no separate login.

Integration with MediaWiki’s Single User Login via OAuth – no separate login.

There is a preview when writing comments.
The initial description can be edited and updated like a summary while the discussion on a task evolves.
Users have a profile showing their latest activity.
There’s a global activity feed.
There is a notification panel on top.
The UI looks modern and works pretty well on devices with small screens.
Tasks can have either zero or one assignee. In Bugzilla an assignee must be set even if nobody plans to work on a ticket.
Tasks can have between zero and unlimited projects (such as code bases, sprints, releases, cross-project tags) associated. In Bugzilla, tickets must have exactly one product, exactly one component, exactly one target milestone, and between zero and unlimited cross-project keywords. That also solves Bugzilla’s problem of dealing with branches, e.g. setting several target milestones.
Projects have workboards (a card wall) with columns for planning sprints (Bugzilla only allowed getting lists of items which you cannot directly interact with from the list view.). Thanks to Wikimedia Deutschland we now also have burndown charts for sprint projects.

The workboard of the Wikimedia Phabricator project, right after the Bugzilla migration.

Burndown chart for a two week sprint of the Wikimedia Analytics team.

Burndown chart for a two week sprint of the Wikimedia Analytics team.

Disadvantages

From a bugmaster point of view there are also small disadvantages:
Some searches are not possible anymore via the web interface, e.g. searching for open tasks which have the same assignee set for more than 12 months ("cookie-licking") or tasks that have been closed within the last month.
Phabricator is more atomic when it comes to actions: I receive more mail notifications and it also takes me slightly longer to perform several steps in a single ticket (though my local Greasemonkey script saves me a little bit of time).

Furthermore, admins don’t have the same powers as in Bugzilla. The UI feels very clean though (breadcrumbs!):

Administrator view for settings policies in Maniphest.

Administrator view for settings policies in Phabricator.

New territories

Apart from the previous list of unexpected situations while migrating, there were also further issues we experienced before or after the migration.
Mass-importing huge amounts of data from an external system into Phabricator was new territory. For example, Phabricator initially had no API to create new projects or to import tickets from other systems. No Phabricator instance with >70000 tasks had existed before – before the migration we had a crash affecting anonymous users and after the migration the reports/statistics functionality became inaccessible (timing out). Those Phabricator issues were quickly fixed by upstream.
And of course in hindsight, there are always a few more things that you would have approached differently.

Next steps

All in all and so far, things work surprisingly well.
We are still consolidating good practices and guidelines for project management (we had a Hangout video session on December 11th about that), I’ve shared some queries helpful for triagers, and we keep improving our Phabricator and bug management related help and documentation. The workflow offered by Phabricator also creates interesting new questions to discuss. Just one example: When a task has several code related projects assigned that belong to different teams, who decides on the priority level of the task?

Next on the list is to replace RT (mostly used by the Operations team) and helping teams to migrate from Trello and Mingle (Language Engineering, Multimedia and parts of Analytics have already succeeded). In 2015 we plan to migrate code repository browsing from gitblit and code review from Gerrit.

OMG we made it

A huge huge thanks to my team: Chase (Operations), Mukunda (Platform), Quim (Engineering Community Team), the many people who contributed code or helped out (Christopher, Daniel, Sean, Valhallasw, Yuvi, and more that I’ve likely forgotten), and the even more people who provided input and opinions (developers, product managers, release management, triagers, bug reporters, …) leading to decisions.
I can only repeat that the upstream Phabricator team (especially Evan) have been extremely responsive and helpful by providing feedback incredibly fast, fixing many of our requests and being available when we ran into problems we could not easily solve ourselves.

Cheers!

Wikimedia in Google Code-in 2014: The first week

Wednesday, December 10th, 2014

Wikimedia takes part in Google Code-in (GCI) 2014. The contest has been running for one week and students have already resolved 35 Wikimedia tasks. You can help making that more (see below).

Google Code-in 2014

Some of the achievements:

  • Citoid offers export in BibTeX format (and more contributions)
  • Analytics’ Dashiki has a mobile-friendlier view
  • Echo‘s badge label text has better readability; Echo uses the standard gear icon for preferences
  • Wikidata’s Wikibase API modules use i18n for help/docs
  • Two MediaWiki extensions received patches to not use deprecated i18n functions anymore
  • MediaWiki displays an error when trying to create a self-redirect
  • The sidebar group separator in MediaWiki’s Installer looks like in Vector
  • The Wikimedia Phabricator docs have video screencasts and an updated Bug report life cycle diagram
  • Huggle‘s on-wiki docs were updated; exceptions received cleanup
  • Pywikibot‘s replicate_wiki supports global args; optparse was replaced by argparse
  • Reasons for MediaWiki sites listed as defunct on WikiApiary were researched
  • Wikimedia received logo proposals for the European Wikimedia Hackathon 2015
  • …and many more.

Sounds good? Want to help? Then please spend five minutes to go through the tasks on your to-do list and identify simple ones to help more young people contribute! Got an idea for a task? Become a mentor!

Wikimedia in Google Code-in 2014

Monday, December 1st, 2014

At work I spent the last weeks mostly working on preparing Wikimedia’s move from Bugzilla to Phabricator which successfully happened last weekend after ~7 months of planning (worth another blog post) and preparing Wikimedia’s participation in Google Code-in (GCI) 2014 (specific steps are listed in the corresponding task in Phabricator). Eventually, Wikimedia got accepted by Google as one of the twelve Free and Open Source organizations taking part in GCI.

Google Code-in 2014

GCI is a contest for 13-17 year old students and starts on December 1st. Students are offered tasks that should take an experienced contributor (knowing the toolchain and infrastructure) up to four hours. Tasks can be about code, documentation, QA, training, outreach, user interface and research. Wikimedia has a central planning and information wikipage.

Last year Wikimedia ended up with 273 tasks, 46 students, and approximately 30 mentors in GCI. The results were above our expectations (as I was initially sceptical if Wikimedia had enough resources). Based on the common questions that students had we improved our onboarding documentation for new contributors (not only GCI students) and our instructions for GCI mentors. Hence Wikimedia is better prepared than last year: So far we have approximately 200 tasks available (not all of them will be available to students immediately) and 24 mentors. Still we need way more tasks and mentors for the next six weeks!

Become a mentor!

If you are active in the Wikimedia community, have that small task on your To-Do list that you haven’t managed to work on yet, and can imagine mentoring a student: Check out our mentor’s corner, register, and create your task!

Wikimedia to migrate its Product Management Tools to Phabricator

Monday, May 19th, 2014

Slightly more correct title might be “Wikimedia to migrate its software development product/project code review issue tracking management planning tools to Phabricator”. Or something like that.

The problem

The Wikimedia technical community has used plenty of different tools for tracking bugs / product management / project management / todo lists. Bugzilla, RT, Gerrit, Mingle, Trello, Scrumbugs, Google Docs, to mention most of them.
From my personal point of view, Wikimedia Foundation is pretty bottom-up: Each team can experiment and use those tools they prefer and suit them. That also means teams might have moved from Bugzilla to Trello to Mingle to Google Docs while other teams prefer(ed) other tools. Bugzilla is our public issue tracker but misses a lot of functionality when it comes to agile development workflows, design review work, or activity feeds.

We also have some connectivity between these tools. Bingle/Bugello to sync some parts between Bugzilla and Mingle/Trello, or its-bugzilla (previously “bugzilla-hooks”) to have Gerrit post comments in Bugzilla tickets about related patches (if the bug number was correctly refered in the commit message). But things are brittle – for example, just this Friday the Gerrit→Bugzilla notifications broke.
All in all, the multitude of tools and channels is not helpful for cross-team collaboration, keeping track of what’s happening, and transparency of discussions and decisions in general as things are discussed in several places.

The idea

In late 2013, the idea was to start a discussion about a possible agreement on a recommendation for a smaller set of tools that teams could agree upon. My colleague Guillaume and I had the pleasure to facilitate the discussion and to ensure it doesn’t remain an idea only. References were a previous evaluation attempt in 2009/2010 and the Gerrit evaluation in 2012.

The steps

The first step was asking interested teams and individuals to describe their needs and workflows on a wiki discussion page.
Its content was then consolidated by Guillaume and cleaned up a little bit more by me. (I felt reminded of GNOME’s decision process to migrate from Subversion to Git but that was a survey among GNOME foundation members and hence a very different approach.)

After having those (sometimes contractive) needs and workflows collected, we tried to decrease the items in the list of candidate tools to consider, plus investigate and encourage discussing them on the related discussion page. For candidate tools not having an online test instance offered on the project’s homepage we wondered whether to set up test instances on Wikimedia Labs to make testing easier, but we left it to anybody strongly favoring a tool to set up that tool. Wikimedia Deutschland already had Scrumbugs instance in production (Scrum on top of Bugzilla) we could point to, and for Phabricator somebody had set up a test instance in Wikimedia Labs already.

To gather the broader community opinion and broader support for investigating more potential (wo)manpower, we started to prepare a Request for comments (RFC). While we listed several options at the beginning (Keep the status quo; status quo+Fulcrum; status quo+Scrumbugs; move completely to Phabricator; move partially to Phabricator; move to GitLab) the feedback quickly turned this into one question to ask the community: Move to Phabricator?
We ran this RFC for three weeks until May 6th and announced it widely on mailing lists and via banners on top of Bugzilla and mediawiki.org.

Parallel to running the RFC, we were working on sorting out the blockers for a potential migration from Bugzilla and documenting things. My colleague Quim created a comparison page between Phabricator and Bugzilla.

The decision

Phabricator project logo

The result of the RFC is that there seems to be general support for moving from our infrastructure tools to Phabricator. This won’t all happen at the same time though – we will start investigating replacing Bugzilla, RT, Trello, Mingle.
For the code review functionality (currently done via Gerrit), more work in Phabricator is needed to fit the needs defined by our community, for example when it comes to Continuous Integration. We do not plan to switch off Gerrit on the day we start using Phabricator in production and we got more items to sort out (see the list of code review related items).

For managing the project to move to Phabricator we use the Phabricator test instance itself (dogfooding for the win), by tracking missing features compared to our existing tools, and tasks that need to be solved for the migration. Also, we asked users of existing tools what they would specifically miss in Phabricator by creating (sub)tasks in our Phabricator test instance.

We have not created a Phabricator production instance yet to which we would potentially migrate to, because in the past Wikimedia ended up with a lot of tools by not enforcing migration.

Spreading the word and resource allocation

For the last months we also ran IRC office hours every two or three weeks in order to discuss and answer questions related to the project.

Last weekend the Wikimedia Hackathon took place in Zürich. There were several Phabricator related sessions (videos available for the sake of transparency; the first two videos are more like discussions though):

  1. A general quick introduction to Phabricator by my colleague Shahyar
  2. A discussion among ~10 people of what needs to be sorted out and done for Day 1 of Phabricator in production (being tracked in this Phabricator project board), plus related stakeholders and who’s going to work on what.
  3. A session by Shahyar specifically explaining the code review concept and tool in Phabricator (remember: we do not plan to switch from Gerrit to Phabricator for code review on the first day)

Current status and next steps

  • General and central information page about Wikimedia’s Phabricator instance.
  • Planning board for Wikimedia Phabricator Day 1 in Production (You see several columns here: ‘Need discussion’ needs more thoughts first until we know how to solve those tasks; ‘Ready to Go’ are tasks for which the way forward is clearly defined and somebody could start working on them; ‘Waiting for upstream’ are tickets that we have reported to upstream; ‘Doing’ is what is currently actively being worked on.)
  • The team working on Wikimedia Phabricator and the stakeholders are defined. We have tasks such as investigating migrating Bugzilla and RT data to Phabricator, plus implementing missing functionality like restricting access to certain projects by default (like for Bugzilla’s “Security” product or RT’s procurement queue) via namespaces. The WMF Technical Operations team will set up low-level infrastructure and puppetize our Phabricator instance. We have stakeholders defined to keep in contact with the team of Product Managers and the Platform and Release Engineering/QA team, we have more interested stakeholders which will provide input to help making decisions and driving things forward, and I am happy to also see volunteers being interested to get involved by diving into the code.
  • To track the tasks that Wikimedia is interested in in Phabricator’s upstream task tracker, we have a “Wikimedia” project in upstream Phabricator and the corresponding planning board.

The usual disclaimer: Plans might be subject to change and there is intentionally no timeline yet.

How you can help

Check out the Get involved section on the central project page and the planning board for Wikimedia Phabricator Day 1 in Production if there are tasks that interest you!

Thanks

I would like to thank everybody in the community who has provided input, help and support. Upstream Phabricator developers have been extremely responsive and interested in discussing our needs and fixing issues – it’s a great pleasure to work with them.
Furthermore, getting to this point would have been impossible without my wonderful colleagues in the Wikimedia Engineering Community Team who have helped so much with communication, prioritization, planning, support.

Wikimedia’s Road to Bugzilla 4.4

Thursday, March 6th, 2014

(How we puppetized, upgraded and moved Bugzilla to another server)

Though we currently also evaluate Wikimedia’s project management tools, we will have to stick with our current infrastructure for a while. Among many other tasks, I spent the last months preparing the upgrade of Wikimedia’s Bugzilla instance from 4.2 to 4.4. Some reasons for upgrading can be found in this Bugzilla comment.

In late November 2013 I started by cleaning up Wikimedia Bugzilla’s custom CSS which was copied about five years ago and not kept in sync. It turned out that 16 of 22 files could be removed as there was no sufficient difference to upstream’s default CSS code (Bugzilla falls back to loading the default CSS file from /skins/default if no custom CSS file is found in /skins/custom). Less noise and less diffing required for future upgrades. In theory.

After testing these CSS changes on a Wikimedia Labs instance and merging them into our 4.2 production instance, I created numerous patches and put them into Gerrit (Wikimedia’s code review tool) by diffing upstream 4.2 code, upstream 4.4 code, and our custom code.

At the same time, Wikimedia’s Technical Operations team wanted to move the Bugzilla server from the kaulen server in our old Tampa datacenter to the zirconium server in our new Ashburn (Eqiad) datacenter. While you’d normally prefer to do only one thing at a time, Daniel Zahn (of Technical Operations) and I decided to create a fresh Bugzilla 4.4 instance from scratch on the new server to see into which problems we would run. During this process Daniel Zahn turned the old setup on kaulen, which was largely manual and had organically grown over the years, into a proper Puppet module. For every “missing module” error we ran into we avoided installing anything from Perl’s CPAN in Bugzilla’s /lib folder and ensured we just rely on distribution packages, for a much cleaner install. Daniel Zahn installed the needed packages by adding them to puppet code. While doing this we also removed Bugzilla’s Sitemap extension as it created sporadic Search::Sitemap errors when running Bugzilla’s checksetup.pl (plus it’s unmaintained anyway). Furthermore I ran into another runtime error to fix.

After fixing all checksetup.pl issues and having Bugzilla accessible via a web browser, only Bugzilla’s upstream CSS was displayed instead of our custom CSS. Neither was Wikimedia’s custom CSS offered as an option in the browser, nor could I log into the new Bugzilla (to check which theme is set as default in the admin settings) as the database dump we used for testing predated the creation of my user account.

After Sean Pringle of Technical Operations deployed a more recent Bugzilla database dump I expected further problems due to upstream changes to CSS loading. I was happy to see that I had been wrong: No problems with our custom CSS theming anymore. Instead, I ran into problems with our custom “See Also” field changes: Adding and removing such URLs triggered errors, and URLs themselves were not displayed (but their corresponding “Remove” checkbox). Thanks to upstream help in #bugzilla on Mozilla IRC I finally found out that Perl’s use base instead of use parent was the culprit.

After creating symlinks to /extensions/WeeklyReport/ to avoid 404 errors for the “Weekly Bug Summary” link in the sidebar (our setup is slightly busted) and after fixing two problems with our cronjobs for whining and data collection we agreed on a date to copy the database, do some maintenance work, and switch the DNS entry. This was announced one week in advance by adding a banner to Bugzilla via its announcehtml parameter.

A few hours before the switch on February 12th 2014, Daniel lowered the Time-to-live (TTL) values of the DNS entry of our Bugzilla. When the migration started, I set Bugzilla’s shutdown parameter to make the web UI inaccessible and also the WebService API return a 503 error for the Bingle script that syncs Bugzilla with Wikimedia’s Mingle instance. This was important to make sure that nobody can write anymore to the old database. We updated the IRC channel topic in #wikimedia-tech to tell that Bugzilla is under scheduled maintenance and logged the action in #wikimedia-operations so it got added to Wikimedia’s Server admin log. All in all we had only forgotten two minor things: Our Gerrit integration (a bot adding Gerrit notifications about related patches as comments in Bugzilla) bot was not able to write and got a 503 error back – Chad quickly disabled it. And our Nimsoft watchmouse sent an “ALERT! Bugzilla: Service Temporarily Unavailable” message to the Operations mailing list.

Sean Pringle migrated the old database from db9 in Tampa, to a new database on db1001 in Eqiad. After this was done, Daniel Zahn ran checksetup.pl to apply the scheme upgrades needed for 4.4.

After 30 minutes of testing to make sure everything works as expected we deployed two more custom patches: Showing common queries on the frontpage and making saved reports work. While having the downtime I also switched off bugmail to do some mass-changes without spamming everybody: I merged some version numbers in the “MediaWiki” product to have a shorter Version dropdown, removed the wikibugs-l watcher account from some bug reports as it is unneeded (set as a global watcher in Bugzilla anyway, hence a potential issue if a ticket was moved to a restricted product like “Security” still triggering public bugmail).

A few minutes before the end of the announced downtime of three hours, Daniel switched DNS so the new Bugzilla on the new server became available to the public. A few hours later, to work around isses for clients not supporting SNI, Daniel changed the order in which Apache loads virtual hosts. This ensures that older clients like Internet Explorer on Microsoft Windows XP will always get to see Bugzilla instead of other miscellaneous web services sharing the same hardware. I had also overlooked a small UI issue that I fixed two days later.

Now that all is done, the result can now be seen on bugzilla.wikimedia.org. All steps to upgrade Wikimedia Bugzilla from 4.2 to 4.4 were documented on a wiki page. You can find all of our custom modifications here.

Wikimedia in Google Code-In 2013

Wednesday, January 22nd, 2014

Google Code-In 2013 Logo

Google Code-In (GCI) 2013 is over and the winners have been announced! Congratulations to everybody!

Quim Gil and I organized the participation of Wikimedia in GCI this year. We set up a central wikipage that we pointed students to for recurring general questions. Mostly these were about expectations on communication, how to use IRC, how to set up a development environment and the toolchain (Git, Gerrit, Bugzilla), plus a list of our mentors and their IRC nicknames and timezones (some enthusiastic students welcome being reminded of timezones, plus having patience is one of the good lessons to learn).
We also had a section for mentors explaining how to write good GCI tasks and which sentences and information should be part of every task description.

For the Wikimedia Engineering Community Team, GCI was a helpful lesson to prepare improving our Annoying little bugs landing page for new contributors (still on my TODO list). Students hopefully found interest in contributing to a great FOSS community (I have seen numerous students who continue contributing and are still active after the contest has finished). I hope that Wikimedia will be able to invite the most active students to the Wikimania conference.

We ended up with way more tasks available than expected (and more than 200 successfully finished), had a number of students who really impressed us by code quality (number of patch reviews required) and speed, and Quim managed surprisingly well to convince established developers to become mentors. Also, the nine hours of time difference between Quim and me was a big advantage in order to respond quickly to requests.

All together, GCI was successful and participating in GCI was good decision, contrary to my initial reluctance.
I would like to thank Google and all involved mentors and students for their hard work and for making this possible.

Bugzilla Tips (XI): Reports: Tickets closed last week by resolution

Friday, August 30th, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

This episode goes deeper into Creating reports and tables and Bugzilla’s Advanced Search.

Three weeks ago a development team asked how to get the number of Bugzilla tickets closed in the last seven days by some resolutions (e.g. FIXED, WONTFIX, INVALID, WORKSFORME) while not being interested in the number of tickets closed as DUPLICATE or LATER.

Another usecase for tabular reports in Bugzilla: The vertical axis will display the components which the team maintains and the horizontal axis will display the bug report resolutions:

bugzilla-closedstats1

We manually selected the components of the team in the “Components” list (keep the Control key pressed to make a multi-selection), and selected all resolutions which should be listed in the bug report (see here for the meaning of the “—” resolution). Keep in mind that these resolutions only refer to the current status of the bug reports.

bugzilla-closedstats2

As we only want to know the number of tickets who were resolved in the last week, we need to query for changes of the resolution. This is where the Custom Search comes into play which allows logical combinations of conditions. For our example:

bugzilla-closedstats3

The generated report looks like this:

bugzilla-closedstats4

Though we chose to include INVALID and WONTFIX resolutions in the search criteria above, the table does not have INVALID and WONTFIX columns because no tickets were closed with that resolution in the last week. Similarly there is no row for the “CLDR” component chosen in the search criteria above because no CLDR ticket was resolved in the last week.

Above example explains how to exclude statuses from the table which you are not interested in. If you want all resolutions displayed in the table anyway (and as setting a resolution requires changing the bug status to RESOLVED), you might already guess that there is an easier way by using “Search By Change History”:

bugzilla-closedstats5

Now in case you wonder why setting the resolutions via the custom search is not sufficient and why it is also required to set the current resolution in the “Product / Component / Status / Resolution” lists above as search criteria: It avoids including tickets which already got reopened in the meantime.

If the numbers are slightly different than your expectations, note that until Bugzilla version 4.2, -7d refers to the beginning of the day in the timezone that Bugzilla is set to, not to the last 168 hours. This has changed in Bugzilla 4.4 though, -1d refers to the last -24h while -1ds refers to the start of the day.

Bugzilla Tips (X): Triage helper tools: Greasemonkey scripts

Friday, August 23rd, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

If you read a lot of Bugzilla tickets per day you run into some recurring situations. For example, a bug report might miss sufficient information and you want to point its reporter to a wikipage which explains how to write good bug reports, or you clean up older rotting tickets without enough information and while closing them you want to explain to the reporter why you close the report.

In such situations, having some one-click stock answers (or “canned responses” as others call them) can come in handy in order to save time. I’ve been using several Greasemonkey scripts in my Firefox browser over those years. The two Javascript files which I use in Wikimedia Bugzilla are available to everybody in the code repository. They can be checked out by running the command git clone https://gerrit.wikimedia.org/r/p/wikimedia/bugzilla/triagescripts.git
To use them in Firefox, one has to install Greasemonkey, visit the Git web interface of the repository with Firefox and click the “Raw” links. An installation dialog will open. Note that these scripts only work if you have canconfirm and editbugs permissions in Bugzilla.

This is how it looks after installing the scripts:

bugzilla-triagehelpers-stockanswers

In the picture above, I clicked the “[Close:WorksForMe]” answer. An explanatory command (which also automatically picks up the name of the reporter) is added and the status “RESOLVED WORKSFORME” is set.

As time goes by I’ve added more functionality for my convenience, mostly links to places which are related to functionality exposed in the Bugzilla interface:

bugzilla-triagehelpers

Most useful to me is

  • coloring reporters based on their email addresses (I might spend less attention on a report created by an established developer or employee than a newcomer),
  • coloring the component (MediaWiki has hundreds of extensions, and extensions which are deployed on Wikimedia servers might receive more attention than a non-deployed 3rd party extension),
  • a link to search for other reports created by the same user, and
  • a link to a graph of the priority distribution of tickets for the component (to check how realistically priorities are set – if you only have open tickets with high priority for your component, then something is wrong).

While the code of these Greasemonkey scripts would welcome some cleanup and refactoring, it works for me. Plus today I finally introduced a bunch of boolean variables at the top of the scripts so users can easily define which functionality s/he wants to enable (or not). You are welcome to give it a try (and provide patches if you feel like hacking away).

Of course all this functionality could also be added to the Bugzilla code on the server but I do not want to clutter the Bugzilla user interface even more for everybody by default.
Note that there is also a “proper” upstream Bugzilla extension called Canned Comments available since May 2012 which I have not played with yet.

Bugzilla Tips (IX): Excluding less important reports from search results

Friday, August 16th, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

Sometimes I’d like to exclude certain bug reports from the search results in Bugzilla.
The most common case is excluding enhancement requests (in order to only get “real” bugs) and excluding lowest and unprioritized priorities (the available values for priorities and severity may differ in other Bugzilla instances), to only have more important stuff listed in the results.

Go to Bugzilla’s Advanced Search and select the product/component that you are interested in, as always. In the “Detailed Bug Information” section, select all values which you do not want to exclude from the Severity and Priority fields:

Selecting Priority and Severity values

Bugzilla Tips (VIII):Using flags to track branches and versions

Friday, August 2nd, 2013

This posting is part of a series on small and sometimes not-so-easy-to-discover functionality in Bugzilla that makes developers’ and users’ lifes more comfortable. It’s based on conversations with users and developers in the last months.

This episode covers an aspect of release management: Branches.
Bugzilla’s support for tracking bugs and bug fixes in several branches/versions has been notoriously bad.

I have seen two ways how software projects using Bugzilla handle this: One way is to clone bug reports and use the Version and Target Milestone fields strictly. Hence one bug report only affects one branch (version), and the very same bug is handled in a separate bug report for a different branch/version.

Another way is to use flags. Flags can have four states:

  • ?: Somebody requested a decision.
  • -: The request was refused.
  • +: The request was approved.
  • By default the field is empty, and no decision is required / the bug report is not affected.

Flags in Wikimedia Bugzilla:

Tracking-branches-flags-wm

Still, using flags requires agreements on workflows, for example after setting “+” (approved) the bug report should only be closed as RESOLVED FIXED once the fix has actually been merged into the branch.

Since version 4.4, bugmail includes an “X-Bugzilla-Flags” email header which allows filtering mail on it. Furthermore, in contrast to keywords, flags can be configured to automatically notify certain email addresses whenever such a flag request is set.

Mozilla Bugzilla even has a more complicated custom implementation of this which covers both testing whether a version is affected and whether a version has received a bug fix, allowing more than the four states mentioned above:

Tracking-branches-flags-moz

If you use a different approach to track branches in Bugzilla, let me know in the comments!