[Freeipa-devel] [PROPOSAL] FreeIPA Test Plan Workflow

[Petr Spacek]

On 19.3.2015 10:11, Martin Kosek wrote:
> On 03/19/2015 09:25 AM, Petr Spacek wrote:
>> Hello,
>>
>> I do not much to add to the process itself. After first reading it seems
>> pretty heavyweight but let's try it, it can be refined at any time :-)
> 
> Right, but then we would need to migrate the data about test completion and so
> on - which is more work. So it is much better to define some working now, than
> to change it couple months later.
> 
> We were already trying to invent something as much lightweight as possible,
> this was the minimum new fields we come for to be able to track the test
> coverage and plans. If you have another proposal how to track it better, I
> would love to hear it, really :-)

Sure. For me the main question is when *designing of tests* should start and
how it is synchronized with feature design. Is it done in parallel? Or
sequentially? When the feedback from test designers flows back? Isn't it too late?

Let's discuss ticket workflow like this:
new -> design functionality&tests -> write code&tests -> test run -> closed

IMHO we should have tests *designed* before we start to implement the final
version of the functionality. It may be too late to find out that interface
design is flawed (e.g. from user's point of view) when the feature is fully
implemented and test phase is reached.

Designing/writing tests early could discover things like poor interface design
sooner, when it is still easy to change interfaces. Currently we have 'design'
reviews before the implementation starts but actually designing tests at the
same time would attract more eyes/brains to the feature design phase. We may
call it 'first usability review' if we wish :-)

In my mind, test designers should be first feature users (even virtually) so
the early feedback is crucial.

Note that this approach does not preclude experimental/quick&dirty prototyping
as part of the design phase but it has to be clear that prototype might (and
should!) be thrown away if the first idea wasn't the best one.


If this is too radical:

To me it seems kind on unnatural to separate testing from overall bug state.
Equivalent of ON_QA state in Bugzilla seems more natural to me as it is kind
of weird to claim that ticket is closed/finished before full testing cycle is
finished.

I.e. the ticket could have states like:
new -> assigned -> qe -> closed
"qe" state can be easily skipped if no testing is (deemed to be) necessary.

Then there is the question if we actually need to separate field for QE state
and Test case field. Test case could behave in the same way as Bugzilla link
field:
- empty field - undecided
- 0 (or string "not necessary" or something) - test case is deemed unnecessary
- non-zero link - apparently, a test case exists

It would be more consistent with what we have for Bugzilla links.

-- 
Petr^2 Spacek