[Freeipa-devel] [TEST][patch-0037]Fixes of dnssec tests

[Oleg Fayans]

On 05/06/2016 12:08 PM, Martin Babinsky wrote:
> On 05/06/2016 11:14 AM, Oleg Fayans wrote:
>>
>>
>> On 05/06/2016 09:48 AM, Martin Basti wrote:
>>>
>>>
>>> On 06.05.2016 09:36, Oleg Fayans wrote:
>>>> Tests are finally stable:
>>>>
>>>> ============================= test session starts
>>>> ==============================
>>>> platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
>>>> rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
>>>> plugins: multihost, sourceorder
>>>> collected 8 items
>>>>
>>>> test_integration/test_dnssec.py ........
>>>>
>>>> ========================= 8 passed in 5561.48 seconds
>>>> ==========================
>>>>
>>>>
>>>>
>>>>
>>>>
>>> PATCH 38 LGTM
>>>
>>> PATCH 37 IIRC I refused to accept workaround for this issue when you
>>> send this (almost the same) patch for first time, are you sure that we
>>> want to hide real issues in tests, to just have green color there?
>>>
>>
>> The underlying issue is 7 months old. Latest update in the issue from
>> Peter Spacek is: "I do not have time to investigate this issue now",
>> which means, that it will stay there for unpredictable amount of time
>> more. If we want to have a "green" jenkins that actually tests existing
>> features, we have to accept workarounds for such long-term issues
>>
>>> Martin
>>
> I have never been a big fan of "having a green jenkins whatever it
> takes" but I understand that there are all kinds of pressure on your
> team to deliver 100% stable test results.
> 
> If the test fails, let it fail or, even better, use 'xfail' markers so
> that we know that this test fails and we should investigate.
Then all 8 existing cases would have to be marked as xfailed.

> 
> I fit fails for such a long time, we should really invest some time to
> look for the root cause of failure(s). If the appointed person does not
> have time for this, he/she should be able to negotiate some time
> allocated for the investigation. If you feel that the test failures are
> not getting enough attention from us then you perhaps need to be more
> proactive in the reporting.

I am quite OK if Peter Spacek receives some more time for investigation
of the root cause of the problem. What I am not OK with is having a
perfectly functional testsuite for otherwise working feature, that is
being deferred for months just because we do not approve of issue
workarounds.

> 
> We really should be fixing issues, not adding workarounds so that tests
> pass.
> 

-- 
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.