Like Murray I’m getting increasingly annoyed with ubuntu’s update policy. The whole idea of not shipping fixes for current breakage out of fear that this may cause new breakage is just lame IMO. You could at least look at what’s been fixed in the stable releases and do a risk analysis instead of just saying nay to everything.
Gnome-games
Ubuntu is shipping version 2.20.1 of gnome-games in gutsy. Lots of bugs has been fixed since that version and among these some frequent (seemingly ubuntu-specific) crashers. The update policy of ubuntu means that fixes for these does not reach the users. I guess the frequent crashers might fall under the category “severe regression” and I could file a SRU report and convince ubuntu to include that patch though.
For hardy ubuntu is shipping version 2.20.0.1. That’s right, a version older than what you get in gutsy. This is because gnome-games now depend on GGZ and before they can ship a 2.21.x version, GGZ must first be included in main. And for this to happen a MIR report must be filed.
So, the “stable” version is crashing and the unstable is not getting any testing. And unless something happens soon they will either be shipping an old version for hardy or a so-far-untested version – that they will of course _not_ provide fixes for later… Something is clearly broken in this process.
“just file the damn reports and be done with it already” I hear you say. Well, I prodded some people on IRC and things are in motion as far as I know but this is not the real problem. The problem is that ubuntu-outsiders have to file all sorts of elaborate reports to get some common sense actions to happen in this crazy bureaucracy. If you really must have this system please make sure you have the manpower to check your build queues and the bug-reports and take action yourself. Don’t rely on outsiders to come and tell you that something is wrong.
If this no-bugfix-updates policy is to be continued I think we should stop being dishonest to our users and extend the template bugzilla comments like the “dupe+fixed”:
“[…] You may want to check for a software upgrade.”
and add
“Unless you are running ubuntu. Ubuntu deliberately holds back bugfixes out of fear that the fix may introduce new bugs. If you want to see this bug fixed in ubuntu please go fill out an SRU report at bla bla bla”
The same problem exist with Deskbar-Applet. Some very annoying bugs where fixed since 2.20.1. In order to avoid spending hours marking duplicates a lot of bugs have been added to the auto-reject list.
I just don’t understand that policy either.
I wish we could add the ones for gnome-games to the auto-reject list too. Unfortunately the stacktraces are too short for that. We get multiple duplicate bugs a day and the whole situation sucks hard. We are wasting both the developers and the users time and each crash is another nutch downwards for the rep of gnome-games.
If the bugfix releases we do is not what the distributions want then we should have some dialog to fix this. Simply ignoring the bugfix release we put out is just plain dumb.
Oh, good. I see I’m not the only one with the problem. I’ve been fighting with it for the past few months… it’s really irritating.
At this point, I’m definitely going to switch to some other distro, although I haven’t decided which. Ubuntu’s lost my confidence and trust.
Does anyone know if Fedora has this problem?
Well, Hardy has had 2.21.5 since the 16th so I think that part is incorrect. It’s incorrect to say “Hardy is shipping” when it’s not been released yet. An SRU has already been done for gnome-games in Gutsy, that is how we got 2.20.1 in the first place. Gutsy was released with 2.20.0.1.
Now, I can appreciate the fact that it’s frustrating to do a bug-fix release and not have distros pick it up immediately. However, from a distro point of view it’s also frustrating to have upstream’s release and then come back *after* the release and complain about not including the latest stuff.
Bottom-line, Ubuntu doesn’t have enough “resources” (read: enough eyes and hands) to keep up with all bug fixes in all upstreams while at the same time keeping a stable release and working on the development of a new one. Ubuntu is mostly composed of volunteers and it’s quite difficult to catch everything so we do have to rely on upstreams (either Debian or upstream upstreams like Gnome) to some extent.
The current SRU process is fairly strict and that came from experiences where updates inadvertently caused other problems. It was decided that it is better to be tougher on what constitutes an SRU and have significant testing. However, many updates *do* take place and we do try to get as much in as we can.
Note, I’m just a volunteer Ubuntu dev so I can speak for Canonical, but I am on the Universe SRU team and I have only rejected 1 SRU request so far and have seen dozens approved.
I feel bad about your experience, I really do, but I’m not sure what exactly we can do at this time to improve the situation. We often re-evaluate our policies and are open to changes, but we are trying to strike a balance between getting important fixes in and keeping a stable release stable.
David B,
Fedora typically doesn’t have the problem ( I wouldn’t say ever) since the goal there is to stay close to upstream releases as much as possible both in terms of patches as well as version numbers. We don’t merely provide security fixes but also new upstream releases including feature enhancements as updates
http://fedoraproject.org/wiki/Objectives
The balance in trying to get new fixes as soon as possible to end users and trying to avoid new regressions is a hard problem and I can somewhat sympathize with what Ubuntu and other distributions are trying to do too.
I have a feeling that Ubuntu’s reluctance to incorporate fixes into releases stems from the broken X pushed in Dapper a couple of years ago that fixed a couple of issues but broke X for a large number of users.
Yes, the reluctance is due to that broken version of X that was published. But gnome-games and Glom are obviously lower risk than X. Not 100% risk-free, but much lower risk.
Really? Synaptic still says 2.20.1 for me now. (64 bit) Looking at launchpad i see that it lists 2.21.5 as of the 16th. I apologize if this is true for non-64 bit.
I understand ubuntu had to do something based on what happened with X but just ignoring bugfixes as default is the worst thinkable way of dealing with it.
Surely there is a different risk in updating some games compared to updating something like X. If there are issues with the quality of the bugfix releases from GNOME it would be far more constructive to start a dialog about how we can fix this. Doing these releases is a complete waste of our time if distributions are not going to pick them up.
So let’s talk about how we can get a process going that ensures that GNOME’s bugfixes are trusted enough.
Well, it appears we’re both right and wrong 😉 The 2.21.5 source is uploaded but it’s not built because it is indeed waiting on ggz. Hopefully that’ll get fixed soon.
The X thing was probably the biggest issue we’ve had, but it’s not the only one. There are certainly different levels of risk associated with different packages, and perhaps Ubuntu doesn’t handle that as well.
There are two aspects to post-release updates. First you need to figure out if you should fix something. Ubuntu has set a blanket “high-impact” criteria for that. Some distros may have a lower threshold. Secondly, you need to decide whether to backport the fix or upgrade the whole package. Ubuntu takes the former approach and from looking at the Fedora page Rahul linked to Fedora takes the latter approach.
Again I would say that resources are a big issue for Ubuntu. It is a very popular distro but has a relatively few developers. I really don’t think that Ubuntu simply ignores bug fixes, but often doesn’t have the manpower deal with them or knowledge of them in the first place. This is especially true in Universe, where it is entirely run by volunteers.
I think it would be great to have a dialog about how we can better address both upstream and user concerns when it comes to stable release updates. Blog comments probably aren’t the best forum for such discussions though. 🙂
just to be clear, as somebody just pointed out to me on IRC, in my first comment where I said “Note, I’m just a volunteer Ubuntu dev so I can speak for Canonical,” it should of course be “I can’t speak for Canonical” 😉
This is why I liked Gentoo (apart from all the waiting around) – because they don’t really do releases, they just update packages into unstable, then move them to stable as they get proven to be so. You’re always on recent versions.
I’ve been impressed so far with Foresight Linux, who have a similar ‘rolling release’ policy, although I’m using 2 alpha 3, so my experience hasn’t been the best! Still, the conary package manager’s got a lot going for it, and I’m hoping this will solve my woes. I was previously with Ubuntu, but monodevelop was really old and broken and of course there’s no chance of getting a new version without upgrading to Hardy, which seemed like a bad idea (although then I go install Foresight 2 alpha, so I guess maybe Hardy wasn’t so far out of reach after all – although that said, Foresight 2 doesn’t have any GNOME 2.21 packages in it).
I had a hard time believing that it actually was like this when I moved to Ubuntu in the first place. I always figured one of the upsides of open source was a sort of division of labor, that is bugs are fixed upstream -> packaged by distros -> used by end-users who file bug reports, and so on.
Right now I’m considering a move to fedora to have proper bug fixes show up. I really hope this behaviour will change sometime soon. At least the canonical guys should ponder it.
Well, right now Ubuntu has over 700 updated .debs (including translation updates it’d be about 900) from 118 source packages in gutsy-updates. I would hardly say that nothing is getting updated. It’s more that our current method is to primarily do non-critical updating in the current development release and have users get the newer software when upgrading to a new stable release.
Ubuntu also have -backports repos where packages from the current development release (Hardy) are built for stable releases (Gutsy, etc.).
Besides what Jordan has said, I’d like to also point out that for stable releases of GNOME stuff (should include gnome-games and deskbar-applet, not sure about glom), I heard there is a blanket permission for SRU. Even there isn’t such a blanket permission, getting individual SRU permissions for GNOME stable releases should be easy. There just need to be somebody who prepare a new package, test it, and go through the SRU process. Only filing a bug asking for updates is probably not going to get much response, since there is just not enough manpower, and many people are focusing on the development release.
In summary, I think this is mainly a resource problem, not a policy problem. If there are people who use Ubuntu stable release and want to see most recent GNOME stable updates in it, I suggest they contact Ubuntu desktop team (https://wiki.ubuntu.com/DesktopTeam) and ask what they can help. I’m rather sure the answer will not be “no, we don’t update GNOME in stable releases”.
I understand that you have limited resources but so do we. The problem with the current process is that while it may save you work it dumps it directly on us.
I already spend a fair amount of time putting together bugfix releases. What you are basically telling me is to go stick that release up my ass and go file a SRU if I think it’s so important. It’s just arrogant beyond belief.
If you want to do things this way _you_ should be carrying the extra load. You go through the release notes and file relevant bureaucratic forms for whatever looks good. You wade through bugzilla and file all the extra duplicates. _You_ should do it – not me.
The real losers here are the end users. We are really doing them a great disservice this way. We hold back fixes for their current problems and waste their time by having them file bug reports for issues that have long since been fixed. This particular gnome-games bug I mentioned is nearing 500 duplicates and still going strong. We keep telling the reporters that the issues has been fixed. “You may want to check for a software upgrade”. Right. If we are to expect our users to keep filing bug reports this is not how we should treat them.
(All this is of course also just my opinion)
To phomes:
First let me clarify that I don’t speak for Canonical or Ubuntu Desktop Team. Like Jordan, I am just a volunteer involved in Ubuntu packaging (an MOTU for those who are familiar with Ubuntu terms), and the areas I am involved in is not about GNOME packages. Like you, I am also just stating my personal opinion.
I was not saying you (which I assume you mean upstream) should bear the burden of doing stable SRUs. I was trying to say that it’s not the policy that prevented an gnome-games update in gutsy, at least it’s not only the policy. You can argue that the policy is unnecessarily bereaucratic that it makes an SRU more hard than it should be, but it’s still far from “Ubuntu deliberately holds back bugfixes” as the blog post says. (And yes, I understand it’s a rant out of frustration, I’m not blaming Thomas for saying it, I just want to clarify the situation a bit.)
In my opinion, “deliberatly hold back bugfixes” is not true unless an SRU is proposed and then rejected. The current situation is more like “Ubuntu’s update policy makes it hard to fix bugs and hurts end users”.
I am most certainly *not* telling you “to go stick that release up [your] ass”. I’m trying to say that because we do have the extra load, which I fully accept is *our* load, it’s not always going to be done immediately or the way upstreams want. We have to have eyes and hands to do it, and we run short a lot of times. Most distros that have a release schedule (as opposed to rolling releases like Gentoo) simply cannot put all new releases into previous releases. Ubuntu has decided that to best serve its users with the resources available we need to do only high-impact/low risk updates for -updates.
The nature of the current distro model is that it does take an amount of time between when upstream authors fix bugs and when end users get them. The goal with Ubuntu is that that time be no more than 6 months (Hardy, for instance should include the work you’ve put into gnome-games since Gutsy was released). For high-impact bugs we make the wait time less, and for security bugs generally even less then that.
If you bug causes a regression from previous Ubuntu releases and has that many dups it sounds like at least a good candidate for an SRU.
(This is just my opinion as well, Sebastien Bacher and the Ubuntu Desktop Team are much better able to help with the specifics)
On a more constructive note:
If it would help you I could specify in the release notes what fixes are very important. This way you can easily identify what you want to pick up. Filing SRU’s is something you’ll have to do yourself though (That thing was designed to discourage anyone from ever filing it). I don’t want to add distribution specific steps in my release process. There is plenty of work involved already.
I also believe that a project like ubuntu will not have a hard time drumming up a testing team. Package the bugfix releases and have them run it for say 2 weeks. If there are no complaints in this time frame it should be fairly safe to ship it to all users. You already have a “pre-released updates” so I guess you already do something like this. I’m sure there are plenty of users out there that would love to get the updates two weeks prior to everybody else.
@ming:
I meant “upstream” but I wrote “I” to emphasize that I only speak for myself.
I think “deliberately holds back bugfixes” is quite accurate. You would know that there is a bug and fix if you just cared to look. Either way the wording is not important. What is important is that the user is told so if the fix is not to be expected before the next major release and what steps can be taken to change that.
btw. I have no idea what prevented the gnome-games update in gutsy. Please inform me so we can avoid this from happening again
@jordan
By “you” I meant ubuntu and more specifically the ubuntu update policy. I’m sorry if you felt that I directed that at you personally. That was by no means the intent. I am not a native English speaker so I probably construct my sentences in a way that make them feel personal while they are not meant so. Mea culpa.
Again SRU’s are overly bureaucratic and I will not waste my time filing elaborate paperwork just to get a distribution to accept a bugfix. Ubuntu created this beast and ubuntu should deal with it themselves.
Well, I agree it is our beast and we need to deal with it. I don’t expect you to file SRUs. We’ve tried several iterations of SRU policies trying to find a balance. I personally feel like it’s a lot of bureaucracy, especially for upstreams who are gracious enough to do bug-fix releases for the stable releases.
I do think Ubuntu’s issue is mostly resources related. We do have the -proposed repo, as you pointed out, but even then sometimes we have to wait for quite some time to get 2 people to say “works for me”. I checked on gnome-games and didn’t find any SRU or “Please update” bugs. There are 49 open bugs currently so if any of them are fixed upstream it’d be nice to get them to users.
I personally don’t have much more to offer as I actually have nothing to do with gnome packages in Ubuntu, though this might be motivation to get into it a bit 🙂
The problem is exactly that ubuntu expects upstream to come and file SRU’s and bug reports before getting bugfixes included in the stable release. This is extra paperwork and it should be unnecessary. A bugfix release should be considered as a SRU. The stuff we put in those releases should be exactly what you want have too. If ubuntu does not feel this is the case then we should talk about how can make it so. We spend time making these releases so that you as distributions can get the most stable and bug free packages possible. If you feel that we add too much crazy stuff that you don’t want to push to the users then say so.
I understand that there is a difference between what we want. We want you to ship our finest newest releases and you want to ensure that your distribution remains stable. What we need is to get to a point where you trust our bugfix releases enough to just test them a while and then ship them. If this is too scary for you them something is broken and we should work on making that not scary. A process like this seems ideal to me:
1) new gnome bugfix release
2) ubuntu ships to testing team (update-proposed)
3) no regression reports for 2 weeks
4) ship to all
The current way of driving this by SRU’s and lauchpad bug reports is tedious and for the case of gnome-games at least have proven itself not to work out.
I’m not saying that the way I propose can be used for everything but it does give a good idea that things are not blowing up. High profile apps may need more careful testing and for kernel, X, etc where hardware plays a role you would need a much broader testing. But for those of us that make bugfix releases we need this kind of process. We can’t run around filing SRU/bugs for each distro. It does not scale. We need you to look at the stuff we provide. Not the other way around.
A lot of the stuff that was fixed for 2.20.2 and 2.20.3 was bugs reported directly to our bugzilla (a lot of crashers from ubuntu too). There are no bug reports in launchpad for those. Tell me how we get these fixes into gutsy asap?
phomes, how could i ever disagree with you? 🙂
i’ve been adding comments like
“this is an Ubuntu problem and has been fixed months ago in GNOME (see bug xxxxxx). please complain to Ubuntu at https://bugs.launchpad.net instead, thanks.”
for the last months to bug reports in gnome bugzilla.
e.g. many evolution ubuntu reports in gnome bugzilla are in fact support questions, i have started to close them as NOTABUG and point them to a support forum instead of spending hours on discussing user problems with them (“can’t send mail”).
we don’t necessarily have to discuss the “unless you are running ubuntu” addition in gnome bugsquad, you can add your own local stock responses like i do (just poke me for the greasemonkey script) already for some special cases. if you feel like getting unfriendlier, just do it.
I totally agree with you!
I was even considering changing distro because of this issue…. so i’ve tried out fedora 8 werewolf when it came out, but it didn’t supported my wacom tablet not even like ubuntu 6.10 … oh well…
Moving to Kubuntu didn’t really solved the issue since “it’s ubuntu” and they both have the same policies and … repositories xD
Last summer the then stable Ubuntu decided to that a kernel update was in order. Subsequently my desktop wouldn’t boot anymore.
I’m now running stable Debian on all of my machine.
phomes, I couldn’t agree more with you. I expect the distributions to pick up bug fix releases automatically. You suggestion to add the new release to gusty-proposed for two weeks before shipping it to all users sounds sane to me. I don’t want to file *any* form for *any* distribution to get a bug fix release in.
I can understand that it is a lot of work, especially for the big amount of packages in the universe repo. But I think the main part of the distribution should contain the latest bug fix releases. Usually, the users will benefit from it.
Thanks Andre. I don’t currently run greasemonkey but custom stock responses sure sounds useful. I have had the need for something like that for some time. Do we have that stuff on l.g.o? I just looked and couldn’t find it.
I don’t think we should start getting unfriendly before ubuntu has had a chance to react. Our distributions are out friends and we should seek to resolve these issues in a friendly manner first.
I understand ubuntu’s position. They want a nice formal QA process for updates to ensure they don’t accidentally break their users installs (again). It’s a perfectly reasonable thing to do. If I was them I would love to have that too.
The problems with the current process as I see it (with regard to gnome):
– they don’t actually have the time for it
– we also don’t have the time for it
– there is no trust
– they chose the wrong place to put it
The first two are obvious. They just don’t have enough man power for such a process. So they expect us to come running when something needs to be done. Problem is that we don’t have the time to tend to every distros special needs. I doubt anything is going to change with these points any time soon…
The last two is where we can change things. Currently ubuntu has no trust in the quality of our bug fixes. They are afraid that a fix will introduce a new bug. That’s a valid concern. If there is reason to believe that the bugfixes that come out of gnome are not trustworthy then we have something to work on. If this is a problem for ubuntu then it will also concern all our other distributions. We need to stop doing the QA work at the distributions and get it into the release process of gnome instead. We are all low on man power so let’s cooperate on catching the issues before they leave gnome.
Currently our “QA” is relying on individual maintainers to test that the bug fixes are good before they ship them. Also we encourage maintainers to run distcheck so that various tests will be run too. Maybe we need stricter rules here?
Can we set up our infrastructure to automatically run distcheck and not release unless it runs without problems? Can we set demands for (ideally 100%) resonable_number% code coverage in these test? Can we demand that dogtail/LDTP tests be setup? Let’s set up static analysis and make sure no new problems are introduced (what’s the deal with coverity btw?) Lastly we could spit out a livecd/vw-ware image (hi rPath) and have real users test the fixes for a day or two before we release.
This is all a lot of work but if that is what the distros want then we should provide – with their help. Something like this would require commitment from the distros. Tests are not going to write themselves and we need people to do the tedious real user testing. In the end though this should be a win for all.
If this is still not enough to convince ubuntu that the fix is good then you e.g. do the 2 weeks in -proposed to do extra testing. Another idea would be to add something like a “bugfixes” channel. Deactivate it by default and allow users to activate it on a per application basis. This way we can tell users in bugzilla to go to synaptic and activate “bugfixes” for gnome-games. This way only the users that are affected by the bug will get the fix (and the potential for new regressions…)
If you want to keep the current system you could convince (me at least) to grade the fixes in the release notes like:
[p1] don’t crash on start up (bug xxxxxxx)
[p2] make some_functionality actually work (bug xxxxxxx)
[p5] move some_button one pixel to the right (bug xxxxxxx)
then you can scan the release notes yourself and fill out your SRU’s for the fixes that are above whatever threshold you find reasonable. We just need to formalize what each grade means then. Like p1 == frequent crasher, etc.
Either way let’s talk about it and try to resolve it together. In the end I just want to be able to deliver fixes for the problems that users come with to our bugtracker.
First off, many thanks for pushing out bug fix releases to gnome-games, as our users in Gentoo Linux benefit a lot from these kind of releases, as we have a custom (not policy) to push out upstream bug fix releases if available. Grabbing patches from various unreleased SVN trees would mean lots of time investment from our part, so bug fixes in release form are very good.
This SRU policy reminds me somewhat Microsofts hotmail service, in a weird way. To get your e-mail delivered (package upgrade included) instead of completely ignored and sent to a black hole, you need to file a bunch of applications (SRU’s) to various programs and then just maybe the servers response of “status=sent” will actually match reality. Imagine GMail, Yahoo mail (Fedora, SUSE, Gentoo, Foresight, etc) and so on having the same policy – as you already put it – it doesn’t scale. Sorry for the analogy, Ubuntu folks, but that’s what it reminds me of.
I started work on the MIRs on friday just gone and when I solicited help in #ubuntu-devel I was eventually told that all of the required ggz bits had been moved into main – looking at libggz in http://archive.ubuntu.com/ubuntu/pool/main/libg/ it looks as though this was done halfway between you asking and me starting work on it.
I couldn’t find any ggz related MIRs on the wiki, so I’m presuming that it was done internally.
Either way, it’s in main now so hopefully it shouldn’t be blocking gnome-games being upgraded.