[Freeipa-devel] [TEST][patch-0037]Fixes of dnssec tests

[Petr Spacek]

On 6.5.2016 15:03, Oleg Fayans wrote:
> 
> 
> On 05/06/2016 12:08 PM, Martin Babinsky wrote:
>> On 05/06/2016 11:14 AM, Oleg Fayans wrote:
>>>
>>>
>>> On 05/06/2016 09:48 AM, Martin Basti wrote:
>>>>
>>>>
>>>> On 06.05.2016 09:36, Oleg Fayans wrote:
>>>>> Tests are finally stable:
>>>>>
>>>>> ============================= test session starts
>>>>> ==============================
>>>>> platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
>>>>> rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
>>>>> plugins: multihost, sourceorder
>>>>> collected 8 items
>>>>>
>>>>> test_integration/test_dnssec.py ........
>>>>>
>>>>> ========================= 8 passed in 5561.48 seconds
>>>>> ==========================
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> PATCH 38 LGTM
>>>>
>>>> PATCH 37 IIRC I refused to accept workaround for this issue when you
>>>> send this (almost the same) patch for first time, are you sure that we
>>>> want to hide real issues in tests, to just have green color there?
>>>>
>>>
>>> The underlying issue is 7 months old. Latest update in the issue from
>>> Peter Spacek is: "I do not have time to investigate this issue now",
>>> which means, that it will stay there for unpredictable amount of time
>>> more. If we want to have a "green" jenkins that actually tests existing
>>> features, we have to accept workarounds for such long-term issues
>>>
>>>> Martin
>>>
>> I have never been a big fan of "having a green jenkins whatever it
>> takes" but I understand that there are all kinds of pressure on your
>> team to deliver 100% stable test results.
>>
>> If the test fails, let it fail or, even better, use 'xfail' markers so
>> that we know that this test fails and we should investigate.
> Then all 8 existing cases would have to be marked as xfailed.

That is perfectly fine - the test simply found a bug and we have to fix it.
There is no point in having "green" tests just to have them.

Let me clarify my comment in the ticket that this is not a test blocker:
Red Hat Bugzilla is using this definition of TestBlocker:
	A test blocker is a bug that prevents at least one test (test case) from
being executed.

As far as I can tell this issue does not block anything. The tests execute and
correctly detect a bug.

It would be a TestBlocker if it was e.g. a bug in installer which prevented
the install and thus prevented some other test cases from being executed.

AFAIK this is not the case. Or am I wrong?

>> I fit fails for such a long time, we should really invest some time to
>> look for the root cause of failure(s). If the appointed person does not
>> have time for this, he/she should be able to negotiate some time
>> allocated for the investigation. If you feel that the test failures are
>> not getting enough attention from us then you perhaps need to be more
>> proactive in the reporting.
> 
> I am quite OK if Peter Spacek receives some more time for investigation
> of the root cause of the problem. What I am not OK with is having a
> perfectly functional testsuite for otherwise working feature, that is
> being deferred for months just because we do not approve of issue
> workarounds.

Sorry, I do not understand this. What do you mean?


>> We really should be fixing issues, not adding workarounds so that tests
>> pass.

+1, it is just a matter of priority

-- 
Petr^2 Spacek