[Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow

[Martin Kosek]

On 03/18/2015 08:18 PM, Martin Koci wrote:
> Hi,
> working with Test Plans for 4.2 features I'd like to outline workflow
> for test plans. The main aim is to have something documented and more
> clear. 
> So I'd like to start with track ticket options. For better tracking and
> managing tickets we could consider 3 new fields in the track ticket.

track --> Trac

>  
> - First field for tracking link for test plan/test case which we can
> refer to. We would call this field just "Test". 

I would name it "Test Case", if it would link to Test Case on wiki/other tool.
With "Test", I would rather understand it to be a link to the actual tests.

> - The second field for tracking tester (QE). Let's called this field "QA
> Contact". This field should inform about contributor (QE). That means -
> you can track unassigned bugs (tracks), assigned but not reviewed (the
> work hasn't started yet), and finished ("-", "+" - see below). According
> to this you can also see who is overloaded and who can help to the
> others.

Right now, we have following person-related fields (in form name - label):
reporter - Reported by
reviewer - Patch review by

So for consistency sake, what about:
tester - Test by

> - The third one for tracking states or flags for test coverage.
> Something like "QE Test Coverage" with four states/flags:
> 
> * Review is needed - " "  (review is required for this issue)
> Empty "QE Test Coverage" should mean "Hey, I need to be reviewed and
> considered whether some test is needed or not".
> * Test is not needed - "-" (Test is not required for this issue)
> * Test exists - "+" (Test already exists - QE done)
> * Test in progress - "?" (Test is required and QE {will} works on
> test{s})
>  
> I can imagine some naming instead of flags +,-,?, ,.  Any ideas?

Right, this is the most important one as we will use it which RFEs/tickets we
want to cover with tests and which not. The states make sense (we come up with
them together anyway), +/-/?/ / makes sense, alternatively we can do:

Test Coverage: yes/no/in progress/ /

> In the track ticket should be described the issue or link to design page
> to get all necessary information for coverage. QE changes state "QE Test
> Coverage" (Test in progress - "?") and creates test plan on wiki [1] and
> add this link to "Test" field in the track. Then QE informs
> freeipa-devel list about test plan if it's OK providing the link to test
> plan. If it's OK QE will start work on particular test cases. When QE is
> done then changes state "QE Test Coverage" to "+" (test exists). 

Makes sense for big features, yes. What about smaller features or bug fixes? Is
test plan also required or would it be sufficient to just add that one or two
tests to XMLRPC tests?

> As well I'd like to propose the possibility that ticket will not be
> closed until QE is done with test?

Still not sure, the requirement is that the test would need to be finished
before GA, as the ticket must be in the milestone where the code is released. I
personally do not see a problem with closing the ticket and leaving it with
"Test Coverage: in progress" which would then be changed to "yes", when test is
done. It is easy to filter tickets with unfinished tests, even if they are closed.

> Hope it makes sense to you. 
> Can I get your thoughts on this, please?
> 
> Thanks,
> /koca
> *[1] - http://www.freeipa.org/page/Main_Page
> 
>