thunderbird upgrade - wtf?

[Adam Williamson]

On Wed, 2009-10-14 at 20:31 +0530, Rahul Sundaram wrote:
> On 10/14/2009 08:23 PM, Jesse Keating wrote:
> 
> > And that's a people problem more than a process problem.  If nobody
> > tests it in updates-testing, then how is the maintainer to know that it
> > is problematic?  Certainly not solvable with even more repos for testing
> > content...
> 
> 3 people give positive feedback and the update is automatically pushed
> from updates-testing to updates despite atleast one feedback to the
> contrary at
> 
> https://admin.fedoraproject.org/updates/F11/FEDORA-2009-9911?_csrf_token=b77b748e49c5311eb85031331cb2f6474028d615
> 
> The UI changes certainly would be visible without any user feedback. The
> buttons getting removed from the toolbar as well as smart folders were
> immediately visible within minutes. Anyone with significant amount of
> email would probably run into the indexing issue soon as well.
> 
> Note that the update indicates it is a security issue but doesnt explain
> what the security fix is nor does it indicate what other major changes
> are there. No notes has been entered to assist the testers. I don't
> think the onus can be placed entirely on potential testers to provide
> feedback within a week. That is just finger pointing and doesn't help
> address such problems or even mitigate it.

It's worth looking in more detail at exactly what the feedback was.
Here's some of the feedback which was marked positive, i.e. +1:

"loving the new search stuff"

"Works for me, is it intended behaviour that several buttons including
"Delete" disappear off the main (top) toolbar?"

without those two bits of feedback - which noted what were later
identified as problems with the update, but nevertheless rated it
positively - it wouldn't have been pushed. it only ever hit exactly +3,
never higher. without those comments, it would have hit a max of +1.

so I disagree with the notion that bodhi / updates-testing are useless
("fat lot of good"), and agree with Jesse that the evidence doesn't
support the conclusion that the best fix is to throw more repositories
at the problem.

I'd agree with Jesse's point that it would've been best for the
maintainers to disable the +3-automatic-push for this update, though
hindsight is always 20-20. perhaps we (QA / rel-eng) need to give more
specific advice about when to use it. My perspective would be that
automatic-push should only be used when you're making a small-scale
update which fixes one or two specific issues and does not change
behaviour in other ways. It should not be used for a big update like
this which rolls up many 'fixes' and things which can't strictly be
described as fixes, because you're going to get a situation like this
where it's hard for a simple +1 / -1 from individual testers to judge
the update.

We might also look at providing better instructions for people providing
feedback on exactly when to use the +1, -1 and 0 options. I'll try and
find some time to look at that.

-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net