Proposals for the Updates Testing Procedure

[Warren Togami]

I am glad to see many test updates being announced on fedora-test-list, 
but may I please suggest some improvements to this mechanism.  We also 
need to discuss the eventual creation of policy for the update approval 
process.

Along with each announced test update, I believe it would be crucial to 
include a link to a corresponding Bugzilla report.  While longer 
discussions pertaining to packages can remain on fedora-test-list, all 
pertinent information should be posted in one place within that test 
update's Bugzilla report.  Why?

* Otherwise it is likely that other Bugzilla reports and important 
information related to an update could easily be lost in the mailing 
list noise.
* All pending updates can easily be found by a single Bugzilla database 
query.
* Each report would become a one-stop-shop for information regarding 
that update.  More responsible sysadmins with proper testing procedures 
can read that report to help in their decision to update.


Discussion of the Update Approval Process
=========================================
We also have not yet discussed the fedora.redhat.com update approval 
process.  I suggest the following to begin the necessary discussion. 
Some of the below are similar in concept to the procedures currently 
being used successfully at fedora.us as described in this document:
http://www.fedora.us/wiki/PackageSubmissionQAPolicy

* Backport Patches v. Upgrade Version
Most libraries, core system and server packages should always have 
backport patches.  End-user applications where other packages do not 
depend on them can possibly be upgraded in version after sufficient 
testing in updates-testing.  There are always possible exceptions to the 
rule, although the determination of backport verses upgrade version will 
need to be decided on a case-by-case basis based upon how intrusive the 
change would be. (Yeah this description sucks.  Reply with a better and 
expanded proposal if you care about this.)

* Time-limit to publish where no negative comments are posted within the 
Bugzilla report.  Senior developers reserve the right to hold an update 
  if a good technical reason can be stated.  (Insert more details here.)

* GPG clearsigned messages of approval
For the test update approval process, a sizable number of people who 
matter (this needs further discussion) should post GPG clearsigned 
messages of approval to the Bugzilla report for the pending update 
currently in testing.  Such messages should be posted if they 
technically agree with the patch/upgrade, and they have fully tested the 
functionality of the updated binary RPM and are satisfied that it is 
better than the previous package.  Otherwise dissenting messages about 
functionality brokenness or suggestions for further package improvement 
are posted.

http://www.fedora.us/wiki/QAChecklist
There should be a checklist similar to this one used at fedora.us that 
contributors must go through and say "Passes all checklist items." 
within their report.  This checklist idea has successfully prevented 
many common problems from being published in fedora.us.  Depending on 
the criticalness of the update, the release managers decide when it is 
the appropriate time to publish based upon proper & signed contributor 
feedback.

Before judging the above to be too an "onerous waste of time" as is the 
reaction that many people have, do know that this type of process has 
worked well at fedora.us during this year.  This fedora.us report below 
is a great example of this type of process in action:

https://bugzilla.fedora.us/show_bug.cgi?id=669
Well... something like this report, except a Fedora Core update would 
have fewer package updates during the approval process, and far more 
users saying "gcc-3.3.2-2 works for me".  (It is important that 
clearsigned messages contain unique identifiers like the full package 
name-version-release because messages like "works for me" alone can be 
abused through a replay attack.)

Why GPG?  Wouldn't that be slow and a waste of time?
====================================================
No.  The GPG clearsigned messages over time that collect within many 
reports and mailing list posts help to build a mass of "historical 
evidence" about the reliability and trustability of the advice given by 
a contributor.  Over time this actually improves the efficiency of 
communication in a massively distributed project like one we are trying 
to create because of the following reasons.

1. Developers and users have a way to quickly search a history of a 
contributor's posts and contributions.  That person may then quickly 
form an opinion about the skill level or reliability of the advice giver.
2. GPG signatures make it a lot more likely that the message came from 
the signer... the actual user holding the key.  If that key is ever 
stolen, it would likely be discovered soon enough by other developers or 
the key owner if fraudulent messages are being posted.
3. Documented HOWTO procedure documents in GPG usage and proper use of 
messages give new contributors (developers, testers, etc.) a structured 
way to begin to build credibility and ease their way into the project.
4. The corpus of good GPG signed technical advice builds relationships 
of trust between developers and users.  This could potentially form the 
basis of a developer certification and nomination of community leaders 
in the future.


In order for GPG to become a standard for the project, we must have 
improved documentation with many examples of usage so newbies can 
quickly understand how it works.  Perhaps better tools that interact 
with the X clipboard would help to improve the speed and ease of use of 
GPG clearsigning of messages too.  Can anybody recommend existing tools 
that may help in this?

Warren Togami
warren at togami.com