Lately I have been lacking time to work on upgrading b.g.o to Bugzilla 3.0 (or actually the CVS version). I started again this weekend by looking into why the ‘GNOME version’ and ‘GNOME target milestone’ fields wouldn’t correctly show all the available drop down options.
Due to Bugzilla 3.0 supporting some custom fields (drop down and free text), I had written some upgrade code to convert the existing ‘hacked’ fields into the newly supported custom fields. This to make future upgrades easier to do. However, I couldn’t figure out why those custom fields didn’t have any options (one of those times that you’ve been staring too long at the same code).
The code looked like this (shows the actual part containing the problem):
if (!$dbh->selectrow_array("SELECT COUNT(*) FROM $new_field_name")) {
$dbh->do("INSERT INTO $old_field_name SELECT * FROM $new_field_name");
}
Last Sunday I took a new look at it and noticed the obvious problem. I was doing things with the wrong fields. So I changed it to this:
if (!$dbh->selectrow_array("SELECT COUNT(*) FROM $old_field_name")) {
$dbh->do("INSERT INTO $new_field_name SELECT * FROM $old_field_name");
}
Meaning, insert the drop down options from the old field into the new field instead of the other way around (oops). I was pretty happy to fix this, so I went ahead and tested it. This consists of removing the existing test database, restoring a dump from b.g.o (a few GB), converting the database into UTF-8 (scripts within Bugzilla do this), then letting Bugzilla do the actual conversion (only when this is almost done my code will be executed). This takes around 4-5 hours, during which my computer is very slow (even with ionice and renice). In the end result I would have a correctly working Bugzilla 3.0 database schema. This would allow me to make a new backup.
After the whole conversion was finished, the drop down fields still did not work. Initially I assumed this was due to having different installations and (big whoops) not having the fix in the installation I tested the conversion with. However, it actually was due do another bug. The whole thing annoyed me too much, so instead I looked at reimplementing the ‘developers’ table instead (more on that later).
After I had enough of reimplementing the developers table I looked more closely at the custom fields code. And noticed a ‘!’ that shouldn’t be there (it would only insert the fields if.. there weren’t any fields to insert). But of course, that wasn’t the only problem… I had an explicit check if any drop down values existed before moving them over with an SQL statement that would not care at all if there weren’t any records in the old table to transfer.
My final code looks like this:
$dbh->do("INSERT INTO $new_field_name SELECT * FROM $old_field_name");
It would be so much nicer if I had written that in the first version…
Reimplementing the developers support was interesting as well. In 3.0, Bugzilla allows a group to only edit one product. Pretty closely to what the developers (partly) does. So as part of the conversion, I wrote some code to create a group per product. Allowing only that group to edit the specific product (and a new option to hide bugs just for developers, or developers of a specific product). By (ab)using other standard 3.0 code I can still show comments made by developers, just only by making a few easy template-only changes.
Creating a group per product results in 350+ groups. I was wondering what kind of performance problems 3.0 would have with that (I have found a bunch of performance problems in 3.0). I decided to try and login as a developer and see if I could easily mark another person as a developer of the same product. Two minutes later Bugzilla finally showed my search results (for a user.. I am not talking about query.cgi). Aargh!
Investigating that performance problem was ‘interesting’. The problems I found before usually consisted of either queries that take too long to execute (for whatever reason), or Bugzilla executing thousands of individual queries instead of limiting it to 1-2 (which required a few -planned- design changes). The listing of users, however, was not related to query performance. MySQL spent about 2-3 seconds executing the various queries. The rest was CPU time. It took a while to investigate, but eventually I found some code that was using a list/array where a few hashes should’ve been used instead (resulting in 1000+ lookups for every user returned. and it returned 2000 users.. this in the slow template part of Bugzilla). Fortunately it wasn’t caused by the many groups like I initially assumed. The performance bug first introduced in 2.22.
I still need to commit the custom field fix and the code that creates groups for the developer. My patch to fix the performance problem is currently awaiting review (upstream). I haven’t made much progress on the 3.0 switch yet (huge amount of work left), but I am a bit closer in reimplementing things we had before. It is unfortunate that reimplementing things feels like I am doing nothing.