QA Application XML save formats.

[Toshio Kuratomi]

> Sorry I haven't been following your qa-assistant very closely, I don't
run
> X most of the time. I did download the rpm from sourceforge, and took
a look
> at it. I'm assuming that version doesn't know how to save the .xml
files you
> referred to?
>
Yes.  I wanted to get some feedback on the XML save format before
releasing something that has a format that's just going to be changed in
the next version.  I've put up a snapshot of my current tree here (using
the current DTD):
  http://qa-assistant.sf.net/snaps/
 
It has "save as" functionality but loading is currently broken.
 
> Since you have taken the time to encapsulate the checklist as an XML
file,
> I think we should all start adopting THAT as the canonical reference,
> including going so far as to write an xslt sheet to render it as html
and
> posting it on the website.
>
> I've created a first round xslt stylesheet, which can see the results
of at
> http://www.ilsw.com/~erik/fedoraus.xml
>
> The xslt stylesheet is at
> http://www.ilsw.com/~erik/checklist.xsl
>
I like that idea alot.  Your stylesheet is nice too.  The text of the
various entries could use some going over by others since they're
currently just my interpretation of the wiki and a few other
non-controversial suggestions. They could probably do with a bit of
reformatting as well as I didn't anticipate them being displayed on a
web page.
 
> I also spent some time thinking about how to integrate our code. I'm
going to
> throw out an idea here, and you can see what you think. If you look at
the
> changes I made to the first couple of entries in fedoraus.xml, you can
see I
> added a <script> section, with some simple code in it. My thought is
that the
> best way to enforce a policy is to put the code to do it in the same
place as
> the policy definition, aka the checklist definition.
>
This is a great plan.  I've been trying to think of the best way to add
automated checking into QA Assistant and this definitely beats the pants
off of anything I was thinking of.  I'd call it <test> instead of
<script> as that's more specific to what we're doing but the idea seems
very sound.
 
I'd like to integrate some sort of signature on the checklist though. 
That way people can tell they're about to execute a script that's part
of an official Fedora Checklist, Michael Schwendt's Alternate Fedora
Checklist, or Vader's Join The Darkside Luke Checklist.  Not sure the
best way to achieve this so the programs that utilize the checklist can
check it, though.
 
> I also created a simple python script to walk through the checklist,
find the
> scripts, and run them. With some tweaking, I think it could work
admirably
> both for an unattended checker, or integrated into a gui like yours.
The
> script is at http://www.ilsw.com/~erik/runchecklist.py
>
> Outstanding issues:
> We need to agree on a way to define input parameters to pass into
scripts,
> ie things like a gpg keyring file, or the srpm filename, or ??? You
can see
> I started doing this, but I haven't finished the job by any means.
>
Do we want to allow different scripting languages to be used?
 
For most scripting languages we could write the variables directly into
the scriptlets.  Alternately we could pass them as environment variables
and have the script (or shell wrapper) compose them.  Using these
methods we don't need to worry about the order of the arguments, just
what they are named.
 
What things would it be okay for the scriptlet to ask for?  Somehow the
program either has to know or ask the user to supply the information the
script wants in the first place.  Should we have
  <param name="srpm_filename" query="What is the SRPM filename?">
so the program can supply srpm_filename if it already knows it or query
the user for it when the script runs?
 
> We need to agree on an output method. I think the easy way to do it is
by
> mapping return codes to states like I show in the example, or taking
the
> entire textual output of the program and matching it to a state name.
We can
> do both, they're both easy.
>
Return value is good for most things but sometimes the scriptlet could
return a better output message than the program would otherwise. I don't
want to teach the program what to do for each individual case in the
checklist, however.  Should we have a tag specify if it returns a
message?  <message>True</message>
Should such scripts output the message as the last line?
 
(We could have all scriptlets be in python and run them in-process so we
can retrieve complex return values directly....)
 
> Other comments:
> I'm not sure I entirely understand the difference between qasave.dtd
and
> checklist.dtd. My personal preference is to generate the qasave dtd
(xml
> schema would be better) and xml directly from the checklist xml. Then
there
> is 1 and only 1 definition of the policy, description, and storage
format.
>
Could definitely be done.  Frugality of disk space led me to define two
separate formats but now that I've implemented it once I think it adds
unnecessary complexity.

My thought on creating one format are that we'd have a base checklist
that merged the checklist.dtd and qasave.dtd information together.
The resolution would be set to 'Needs-Reviewing' for all entries.  When
a program ran through a checklist and wanted to save state it would
write out a complete checklist file that was based on the original but
with changes where the user changed states, edited the output message
for a review item, etc.
 
The properties element could use some reworking here as well.  The base
checklist could specify the properties the checklist supports and the
type each is.
 
[Thought: perhaps all input that the scriptlets could use should be a
checklist property.  Are there times when a scriptlet would need to know
something that wasn't a property?  gpg keyring is really a program
preference for instance... I'd store preferences in GConf and the
scriplets could extract it... but you might not want the command line
script to need GConf (GConf isn't graphical, but it isn't something you
expect a standard command-line app to need....)]
 
> I think the checklist probably needs to carry more of the policy
information
> we've argued about (and not decided on) with it. IE I'd like for each
entry
> to have a "class" or something indicating if it was a Blocker,
Non-Blocker,
> Important, or Cosmetic only.
>
Each checklist item has a list of allowable resolution states:
  Needs-Reviewing, Pass, Fail, Non-Blocker, Not-Applicable.

Each item must have Pass and Needs-Reviewing.  Depending on whether I
thought the item was definitively a Blocker or Minor or could fall into
either category I added other resolutions.  We could change names, add
resolution states, and reassign which set of resolutions are available
for each item pretty easily.  Will that work or do you think we should
have actual "classes" that predefine which set of resolutions are
available?  We'd still need to have the states array as we need to set
the output messages on a per state basis....
 
> I think the checklist xml also needs to carry whatever information
might be
> needed to create a QA report in the affirmative or negative for that
> particular entry. IE you should be able to create some sort of human
friendly
> listing of the packages status by transforming a "checklist data file"
and a
> "checklist" together. Maybe you already have this?
>
If I understand what your asking, this is basically what QA Assistant
does.  (Or will once loading of save files works correctly.)
 
There might be some value to creating a command line tool to do this as
well.  Perhaps the commandline script that runs tests embedded in the
checklist could use xml as its internal representation and then xslt
transform it into a QA Report.  Then, run-checklist.py --notest save.xml
could do the transform on a saved checklist without running any new
tests.

-Toshio
-- 
_______S________U________B________L________I________M________E_______
  t  o  s  h  i  o  +  t  i  k  i  -  l  o  u  n  g  e  .  c  o  m
                                                          GA->ME 1999
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20040607/07d172a8/attachment.sig>