GNOME’s Bugzilla instance is old. Too old.
It will get upgraded in the next weeks.
You should help by playing with the test instance. Go read these instructions and do it!
More information to come. Stay tuned.
GNOME’s Bugzilla instance is old. Too old.
It will get upgraded in the next weeks.
You should help by playing with the test instance. Go read these instructions and do it!
More information to come. Stay tuned.
<tl;dr>: Wikimedia migrated its bug tracking from Bugzilla to Phabricator in late November 2014.
After ten years of using Bugzilla with 73681 tickets and ~20000 user accounts and after months of planning, writing migration code, testing, gathering feedback, discussing, writing more code, writing documentation, communicating, et cetera, Wikimedia switched from Bugzilla to Phabricator as its issue tracking tool.
Phabricator is a
fun adventure game collaboration platform and a forge that consists of several well-integrated applications. Maniphest is the name of the application for handling bug reports and tasks.
My announcement from May 2014 explained the idea (better collaboration and having less tools) and the decision making process that led to choosing Phabricator and starting to work on making it happen.
Quim already published an excellent summary of Wikimedia Phabricator right after the migration from Bugzilla, covering its main features and custom functionality that we implemented for our needs. Read that if you want to get an overview of how Phabricator helps Wikimedia with collaborating and planning in software development.
This blog post instead covers more details of the actual steps taken in the last months and the migration from Bugzilla itself. If you want even more verbose steps and information on the progress, check the status updates that I published every other week with links to the specific tickets and/or commits.
After reviewing our project management tools and closing the RfC the team started to implement a Wikimedia SUL authentication provider (via OAuth) so no separate account is needed, work on an implementation to restrict access to certain tasks (access restrictions are on a task level and not on a project level), and creating an initial Phabricator module in Puppet.
We started to discuss how to convert information in Bugzilla (keywords, products and components, target milestones, versions, custom fields, …), which information to entirely drop (e.g. the severity field, the history of field value changes, votes, …), and which information to only drop as text in the initial description instead of a dedicated field. More information about data migrated is available in a table. This constantly influenced the scope of the script for the actual data migration from Bugzilla (more information on code).
We already had a (now defunct) Phabricator test instance in Wikimedia Labs under fab.wmflabs.org which we now started to also use for planning the actual migration.
There’s a 7 minute video summary from June describing the general problem with our tools that we were trying to solve and the plan at that time. We also started to write help documentation.
As we got closer to launching the final production instance on phabricator.wikimedia.org, we decided to split our planning into three separate projects to have a better overview: Day 1 of a Phabricator Production instance in use, Bugzilla migration, and RT migration.
On September 15th, phabricator.wikimedia.org launched with relevant content imported from the fab.wmflabs.org test instance which we had used for dogfooding. In the case of Wikimedia, this required setting up SNI and making it work with nginx and the certificate to allow using SUL and LDAP for login. After the production instance had launched we also had another Hangout video session to teach the very basics of Phabricator.
To provide a short impression of further stuff happening in the background: Elasticsearch was set up as Phabricator’s search backend, some legal aspects (footer, determining the license of submitted content) were brought up, phab-01.wmflabs.org was set up as a new playground, and we made several further customizations when it comes to user-visible strings and information on user pages within Phabricator. In the larger environment of Wikimedia infrastructure interacting with the issue tracker, areas like IRC bots, interwiki links, on-wiki templates, and automatic notifications in tasks about related patches in the code review system were dealed with or being worked on.
We also had a chicken and egg problem to solve: Accounts versus tickets. Accounts in Bugzilla are defined by email addresses while accounts in Phabricator are user names. For weeks we were asking Bugzilla users and community users to already create an account in Phabricator and “claim” their Bugzilla accounts by entering the email address that they used in Bugzilla in their Phabricator account settings. The plan was to import the tickets and account ‘placeholders’ and then use cron jobs to connect the placeholder accounts with the actual users and to ‘claim’/connect their past Bugzilla contributions and activity by updating the imported data in Phabricator.
On October 23th, we made a separate “bugzillapreview” test instance available on Wikimedia Labs with thousands of Bugzilla tickets imported. For two weeks, the community was invited to check how Bugzilla tickets are going to look in Phabricator after the migration and to identify more potential issues. The input was helpful and valuable: We received 45 reports and fixed 25 of them (9 were duplicates, 2 invalid, and 9 got declined).
Having had reached a good overview, we created a consolidated list of known issues and potential regressions created by the migration from Bugzilla to Phabricator and defined a final date for the migration: November 21-23.
Keeping timestamps of comments intact (such as the original creation date of a ticket in Bugzilla or when a certain comment was made) was still something to sort out at this point (and got tackled). It would have been confusing and would have broken searches that triagers need when trying to clean up (e.g. tickets which have not seen updates for two years).
It was also tricky performance-wise to keep the linear numbering order of reports which was requested by many people to not solely depend on URL redirects from bugzilla.wikimedia.org to phabricator.wikimedia.org which we planned to set up (more information on the redirect setup). As we already had ~1400 tickets in Phabricator we went for the simple rule “report ID in Bugzilla + 2000 = task ID in Phabricator”.
Regarding documentation and communication, we created initial project creation guidelines, sent one email to those 66 users of personal tags in Bugzilla warning that tags will not be migrated, sent two emails to the 850 most recently active Bugzilla users asking them to log into Phabricator and provide their email address used in Bugzilla to claim their imported contributions as part of the migration already (for comparison, the average number of active users per month in Bugzilla was around 500+ for the last months), put migration announcement banners on mediawiki.org and every page on our Bugzilla, sent reminders to the wikitech-l, mediawiki-l, wikitech-ambassadors, and wmfall mailing lists.
After a last ‘Go versus No-Go’ meeting on November 12th, we set up the timeline with the list of steps to perform for the migration, ordered, with assignees defined for each task. This was mostly based on the remaining open dependencies of our planning task. We had two more IRC office hours on November 18 and 19 to answer questions regarding the migration and Phabricator itself.
While migrating, the team used a special X-Forwarded-For header to still be able to access Bugzilla and Phabricator via their browsers while normal users trying to access Phabricator or Bugzilla were redirected to a wikipage telling them what’s going on and where to escalate urgent issues (MediaWiki support desk or IRC) while no issue tracker is available. With aforementioned URL redirects in place we intended to move and keep Bugzilla available for a while under the new address old-bugzilla.wikimedia.org.
The migration started by switching Bugzilla to read-only for good. Users can still log into Bugzilla (now available at old-bugzilla.wikimedia.org) and e.g. run their searched queries or access their list of votes on the outdated data but they cannot create or change any existing tickets.
We pulled phabricator.wikimedia.org and disabled its email interface, switched off the code review notification bot for Bugzilla, and switched off the scripts to sync Bugzilla tickets with Mingle and Trello.
The data migration started by applying a hack to workaround a Bugzilla XML-RPC API issue (see below), running the migration fetch script (tasks and comments), reverting the hack, running the migration create script (attachments), moving Bugzilla to old-bugzilla.wikimedia.org, starting the cron jobs to start assigning Bugzilla activity to Phabricator users by replacing the generic “bzimport” user by the actual corresponding users, and setting up redirects from bugzilla.wikimedia.org URLs.
After several of those data migration steps we performed numerous tests. In parallel we prepared emails and announcements to send out and publish once we’re finished, updated links to Bugzilla by Phabricator on dozens of wikipages, updating MediaWiki templates on the Wikimedia, and further small tasks.
For those being curious about time spans: Fetching the 73681 Bugzilla tickets took ~5 hours, importing them ~25 hours, and claiming the imported user contributions of the single most active Bugzilla user took ~15 minutes.
But obviously we were pioneers that could not rely on Stackoverflow.
Even if you try to test everything, unexpected things happen while you are running the migration. I’m proud to say that we (well, rather Chase, Daniel, Mukunda and Sean when it came to dealing with code) managed to fix all of them. And while you try to plan everything, for such a complex move that nobody has tried before, there are things that you simply forget or have not thought about:
We silently reopened Phabricator on late Sunday evening (UTC) and announced its availability on Monday morning (UTC) to the wikitech-l community and via the aforementioned blogpost.
Phabricator has many advantages compared to Bugzilla: Wikimedia users do not reveal their email addresses and users do not have another separate login and password. (These were the most popular complaints about Bugzilla.)
There is a preview when writing comments.
The initial description can be edited and updated like a summary while the discussion on a task evolves.
Users have a profile showing their latest activity.
There’s a global activity feed.
There is a notification panel on top.
The UI looks modern and works pretty well on devices with small screens.
Tasks can have either zero or one assignee. In Bugzilla an assignee must be set even if nobody plans to work on a ticket.
Tasks can have between zero and unlimited projects (such as code bases, sprints, releases, cross-project tags) associated. In Bugzilla, tickets must have exactly one product, exactly one component, exactly one target milestone, and between zero and unlimited cross-project keywords. That also solves Bugzilla’s problem of dealing with branches, e.g. setting several target milestones.
Projects have workboards (a card wall) with columns for planning sprints (Bugzilla only allowed getting lists of items which you cannot directly interact with from the list view.). Thanks to Wikimedia Deutschland we now also have burndown charts for sprint projects.
From a bugmaster point of view there are also small disadvantages:
Some searches are not possible anymore via the web interface, e.g. searching for open tasks which have the same assignee set for more than 12 months ("cookie-licking") or tasks that have been closed within the last month.
Phabricator is more atomic when it comes to actions: I receive more mail notifications and it also takes me slightly longer to perform several steps in a single ticket (though my local Greasemonkey script saves me a little bit of time).
Furthermore, admins don’t have the same powers as in Bugzilla. The UI feels very clean though (breadcrumbs!):
Apart from the previous list of unexpected situations while migrating, there were also further issues we experienced before or after the migration.
Mass-importing huge amounts of data from an external system into Phabricator was new territory. For example, Phabricator initially had no API to create new projects or to import tickets from other systems. No Phabricator instance with >70000 tasks had existed before – before the migration we had a crash affecting anonymous users and after the migration the reports/statistics functionality became inaccessible (timing out). Those Phabricator issues were quickly fixed by upstream.
And of course in hindsight, there are always a few more things that you would have approached differently.
All in all and so far, things work surprisingly well.
We are still consolidating good practices and guidelines for project management (we had a Hangout video session on December 11th about that), I’ve shared some queries helpful for triagers, and we keep improving our Phabricator and bug management related help and documentation. The workflow offered by Phabricator also creates interesting new questions to discuss. Just one example: When a task has several code related projects assigned that belong to different teams, who decides on the priority level of the task?
Next on the list is to replace RT (mostly used by the Operations team) and helping teams to migrate from Trello and Mingle (Language Engineering, Multimedia and parts of Analytics have already succeeded). In 2015 we plan to migrate code repository browsing from gitblit and code review from Gerrit.
A huge huge thanks to my team: Chase (Operations), Mukunda (Platform), Quim (Engineering Community Team), the many people who contributed code or helped out (Christopher, Daniel, Sean, Valhallasw, Yuvi, and more that I’ve likely forgotten), and the even more people who provided input and opinions (developers, product managers, release management, triagers, bug reporters, …) leading to decisions.
I can only repeat that the upstream Phabricator team (especially Evan) have been extremely responsive and helpful by providing feedback incredibly fast, fixing many of our requests and being available when we ran into problems we could not easily solve ourselves.
Wikimedia takes part in Google Code-in (GCI) 2014. The contest has been running for one week and students have already resolved 35 Wikimedia tasks. You can help making that more (see below).
Some of the achievements:
Sounds good? Want to help? Then please spend five minutes to go through the tasks on your to-do list and identify simple ones to help more young people contribute! Got an idea for a task? Become a mentor!
At work I spent the last weeks mostly working on preparing Wikimedia’s move from Bugzilla to Phabricator which successfully happened last weekend after ~7 months of planning (worth another blog post) and preparing Wikimedia’s participation in Google Code-in (GCI) 2014 (specific steps are listed in the corresponding task in Phabricator). Eventually, Wikimedia got accepted by Google as one of the twelve Free and Open Source organizations taking part in GCI.
GCI is a contest for 13-17 year old students and starts on December 1st. Students are offered tasks that should take an experienced contributor (knowing the toolchain and infrastructure) up to four hours. Tasks can be about code, documentation, QA, training, outreach, user interface and research. Wikimedia has a central planning and information wikipage.
Last year Wikimedia ended up with 273 tasks, 46 students, and approximately 30 mentors in GCI. The results were above our expectations (as I was initially sceptical if Wikimedia had enough resources). Based on the common questions that students had we improved our onboarding documentation for new contributors (not only GCI students) and our instructions for GCI mentors. Hence Wikimedia is better prepared than last year: So far we have approximately 200 tasks available (not all of them will be available to students immediately) and 24 mentors. Still we need way more tasks and mentors for the next six weeks!
If you are active in the Wikimedia community, have that small task on your To-Do list that you haven’t managed to work on yet, and can imagine mentoring a student: Check out our mentor’s corner, register, and create your task!
Uhm, no blog post for ages. Work and university kept me way busier (and less energetic to do much other things) than expected.
As announced in May 2014, Wikimedia will replace Bugzilla with Phabricator very soon for issue tracking. The Wikimedia Phabricator production instance is up and running and the migration from Bugzilla and RT is the next step on our migration timeline. A comparison between Bugzilla and Phabricator and documentation are available.
Time to publish my quick notes about installing Phabricator locally on a Fedora 20 machine. Maybe it’s useful for somebody, maybe not.
Phabricator’s official installation guide is pretty wonderful so I’ll only cover what I, being a simple user avoiding to think on his own and just blindly following guidelines, still had to do (which might be very obvious to tech-savvy users). I expect you to know that systemctl restart httpd.service, systemctl restart mysql.service and more /var/log/httpd/error_log are your friends.
In the end, my httpd.conf file looked like this:
<VirtualHost *> ServerName localhost.foo DocumentRoot /var/www/html/phab/phabricator/webroot <Directory "/var/www/html/phab/phabricator/webroot"> Options Indexes FollowSymLinks AllowOverride All Require all granted </Directory> RewriteEngine on RewriteRule ^/rsrc/(.*) - [L,QSA] RewriteRule ^/favicon.ico - [L,QSA] RewriteRule ^/(.*)$ /index.php?__path__=/$1 [B,L,QSA] ErrorLog /var/log/httpd/error_log </VirtualHost>
Being back home and having covered the first two days already, a short overall summary:
GNOME.Asia 2014 conference in Beijing was very well organized – fast and stable WiFi, free water, venue, hotel and sport facilities all within 5 minutes of walking.
While GNOME’s European GUADEC conference is more like meeting lots of old friends (and great new community members) for an old fart like me, GNOME.Asia is about meeting those community members who often cannot make it to the European GUADEC conference (prices of plane tickets), plus spreading the word.
For ignorant Europeans like me, it’s a lot more about listening and learning about the diversity of our community, problems and differences in other areas, cultures, societies. (Plus in this case getting out of that Western internet services bubble of Facebook/Twitter/Youtube/etc. which feel ubiquitous, to see a different bubble yet to explore and understand further.)
Check out the many great photos and the two amazing videos of day 1 and day 2. I’m convinced we won’t have to spend months waiting for the recordings of talks and presentations given either. Now can we import this awesome photo and video team for GUADEC please?
I really hope to see many people at GNOME.Asia 2015 again, and I also hope I’ll find some random reason to go back to Beijing and China soon.
First things first (so you can stop reading if you disagree here):
Outreach programs need to scale and be sustainable to be a long-term success. Is that the case? I’m not sure myself; depends on the criteria you come up with.
As Jim wrote, “the program grew to a size that overwhelmed our administrative staff person” (to simplify it).
Outreach Programs need to run without putting the GNOME Foundation or any other involved entity into financial or legal problems. I don’t follow closely enough, so I wonder if GNOME Foundation and organizations taking part in OPW have considered creating a separate legal entity with a dedicated mission to facilitate more diversity in free and open source software projects (as OPW has grown and is bigger than what GNOME is about)?
Do OPW participants (and GSoC participants) keep participating in our community after OPW is over, and do some become mentors themselves? (And if participants had sufficient time and financial resources, would they also participate if no money was offered?)
I consider the GNOME Documentation project to be a rather successful project keeping OPW’ers involved: Of those 14 OPW participants who worked on GNOME documentation, 6 pretty much vanished after OPW. 2 have mentored, 2 will likely mentor soon, and the rest is still around or around a bit. I don’t have numbers to compare and interpret (Marina posted some) but other GNOME teams feel less successful to me.
I’ve been wondering why. I’m sure others see other criteria and practices and I have not investigated other organizations who take part in OPW – feel free to add your impressions to the comments.
What to consider best practices?
Statistics: Wikimedia has some statistics (beta) on community contributors joining and leaving. Which does not tell you why people join or leave of course. Could organizations be better to find out the “why”?
For example, I kept in contact with my OPW participant after OPW was over, but she got busy with her new job and moving to a new location (socializing). And I don’t want to be too be pushy and expectant towards volunteer contributions. Could I have done something better or differently? I don’t know. We all have our own lives and make our own decisions what to invest our time on.
Thanks to my team (Sumana, Quim, Guillaume) for discussing diversity in general and thanks to Kat for discussing best practices and numbers about OPW in GNOME Documentation.
And obviously everything written here is my personal opinion. As usual.
Yesterday (Friday) before the conference started, Kat, Dave and I held a hands-on session about making your first contribution to GNOME documentation (also see Kat’s blogpost). As a result, a number of tickets in Bugzilla have received comments and patches.
Apart from the pleasure to walk around in the room, take a look over the shoulders of people and helping out on non-obvious things, it was extremely eye-opening to realize again how many obstacles you need to pass in order to finally make a contribution – running a recent GNOME version, finding the documentation that is supposed to explain the following steps, having a GNOME Bugzilla account, finding a task (bug report) that sounds interesting, finding the corresponding code repository, locating the documentation file to patch in that code repository (in some subfolder called “C” instead of “en”), using git (formatting the patch, providing a commit message), uploading the patch somewhere for review.
Today I gave a presentation about managing bug reports in GNOME. About 15 to 20 people attended it and I’m very happy how it turned out – we ran out of time to triage a few tickets at the end, but the audience was interested and asked really good questions (e.g. the upstream-downstream relation)! I’m looking forward to the video recording of it.
I would like to thank the organizers (especially Emily for her endless patience helping me to sort out visa issues), the sponsors, and the GNOME Foundation for paying part of my travel costs.
Slightly more correct title might be “Wikimedia to migrate its software development product/project code review issue tracking management planning tools to Phabricator”. Or something like that.
The Wikimedia technical community has used plenty of different tools for tracking bugs / product management / project management / todo lists. Bugzilla, RT, Gerrit, Mingle, Trello, Scrumbugs, Google Docs, to mention most of them.
From my personal point of view, Wikimedia Foundation is pretty bottom-up: Each team can experiment and use those tools they prefer and suit them. That also means teams might have moved from Bugzilla to Trello to Mingle to Google Docs while other teams prefer(ed) other tools. Bugzilla is our public issue tracker but misses a lot of functionality when it comes to agile development workflows, design review work, or activity feeds.
We also have some connectivity between these tools. Bingle/Bugello to sync some parts between Bugzilla and Mingle/Trello, or its-bugzilla (previously “bugzilla-hooks”) to have Gerrit post comments in Bugzilla tickets about related patches (if the bug number was correctly refered in the commit message). But things are brittle – for example, just this Friday the Gerrit→Bugzilla notifications broke.
All in all, the multitude of tools and channels is not helpful for cross-team collaboration, keeping track of what’s happening, and transparency of discussions and decisions in general as things are discussed in several places.
In late 2013, the idea was to start a discussion about a possible agreement on a recommendation for a smaller set of tools that teams could agree upon. My colleague Guillaume and I had the pleasure to facilitate the discussion and to ensure it doesn’t remain an idea only. References were a previous evaluation attempt in 2009/2010 and the Gerrit evaluation in 2012.
The first step was asking interested teams and individuals to describe their needs and workflows on a wiki discussion page.
Its content was then consolidated by Guillaume and cleaned up a little bit more by me. (I felt reminded of GNOME’s decision process to migrate from Subversion to Git but that was a survey among GNOME foundation members and hence a very different approach.)
After having those (sometimes contractive) needs and workflows collected, we tried to decrease the items in the list of candidate tools to consider, plus investigate and encourage discussing them on the related discussion page. For candidate tools not having an online test instance offered on the project’s homepage we wondered whether to set up test instances on Wikimedia Labs to make testing easier, but we left it to anybody strongly favoring a tool to set up that tool. Wikimedia Deutschland already had Scrumbugs instance in production (Scrum on top of Bugzilla) we could point to, and for Phabricator somebody had set up a test instance in Wikimedia Labs already.
To gather the broader community opinion and broader support for investigating more potential (wo)manpower, we started to prepare a Request for comments (RFC). While we listed several options at the beginning (Keep the status quo; status quo+Fulcrum; status quo+Scrumbugs; move completely to Phabricator; move partially to Phabricator; move to GitLab) the feedback quickly turned this into one question to ask the community: Move to Phabricator?
We ran this RFC for three weeks until May 6th and announced it widely on mailing lists and via banners on top of Bugzilla and mediawiki.org.
Parallel to running the RFC, we were working on sorting out the blockers for a potential migration from Bugzilla and documenting things. My colleague Quim created a comparison page between Phabricator and Bugzilla.
The result of the RFC is that there seems to be general support for moving from our infrastructure tools to Phabricator. This won’t all happen at the same time though – we will start investigating replacing Bugzilla, RT, Trello, Mingle.
For the code review functionality (currently done via Gerrit), more work in Phabricator is needed to fit the needs defined by our community, for example when it comes to Continuous Integration. We do not plan to switch off Gerrit on the day we start using Phabricator in production and we got more items to sort out (see the list of code review related items).
For managing the project to move to Phabricator we use the Phabricator test instance itself (dogfooding for the win), by tracking missing features compared to our existing tools, and tasks that need to be solved for the migration. Also, we asked users of existing tools what they would specifically miss in Phabricator by creating (sub)tasks in our Phabricator test instance.
We have not created a Phabricator production instance yet to which we would potentially migrate to, because in the past Wikimedia ended up with a lot of tools by not enforcing migration.
For the last months we also ran IRC office hours every two or three weeks in order to discuss and answer questions related to the project.
Last weekend the Wikimedia Hackathon took place in Zürich. There were several Phabricator related sessions (videos available for the sake of transparency; the first two videos are more like discussions though):
The usual disclaimer: Plans might be subject to change and there is intentionally no timeline yet.
Check out the Get involved section on the central project page and the planning board for Wikimedia Phabricator Day 1 in Production if there are tasks that interest you!
I would like to thank everybody in the community who has provided input, help and support. Upstream Phabricator developers have been extremely responsive and interested in discussing our needs and fixing issues – it’s a great pleasure to work with them.
Furthermore, getting to this point would have been impossible without my wonderful colleagues in the Wikimedia Engineering Community Team who have helped so much with communication, prioritization, planning, support.