[Freeipa-devel] [TEST][patch-0037]Fixes of dnssec tests

[Oleg Fayans]

On 05/06/2016 03:25 PM, Petr Spacek wrote:
> On 6.5.2016 15:03, Oleg Fayans wrote:
>>
>>
>> On 05/06/2016 12:08 PM, Martin Babinsky wrote:
>>> On 05/06/2016 11:14 AM, Oleg Fayans wrote:
>>>>
>>>>
>>>> On 05/06/2016 09:48 AM, Martin Basti wrote:
>>>>>
>>>>>
>>>>> On 06.05.2016 09:36, Oleg Fayans wrote:
>>>>>> Tests are finally stable:
>>>>>>
>>>>>> ============================= test session starts
>>>>>> ==============================
>>>>>> platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
>>>>>> rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
>>>>>> plugins: multihost, sourceorder
>>>>>> collected 8 items
>>>>>>
>>>>>> test_integration/test_dnssec.py ........
>>>>>>
>>>>>> ========================= 8 passed in 5561.48 seconds
>>>>>> ==========================
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> PATCH 38 LGTM
>>>>>
>>>>> PATCH 37 IIRC I refused to accept workaround for this issue when you
>>>>> send this (almost the same) patch for first time, are you sure that we
>>>>> want to hide real issues in tests, to just have green color there?
>>>>>
>>>>
>>>> The underlying issue is 7 months old. Latest update in the issue from
>>>> Peter Spacek is: "I do not have time to investigate this issue now",
>>>> which means, that it will stay there for unpredictable amount of time
>>>> more. If we want to have a "green" jenkins that actually tests existing
>>>> features, we have to accept workarounds for such long-term issues
>>>>
>>>>> Martin
>>>>
>>> I have never been a big fan of "having a green jenkins whatever it
>>> takes" but I understand that there are all kinds of pressure on your
>>> team to deliver 100% stable test results.
>>>
>>> If the test fails, let it fail or, even better, use 'xfail' markers so
>>> that we know that this test fails and we should investigate.
>> Then all 8 existing cases would have to be marked as xfailed.
> 
> That is perfectly fine - the test simply found a bug and we have to fix it.
> There is no point in having "green" tests just to have them.
> 
> Let me clarify my comment in the ticket that this is not a test blocker:
> Red Hat Bugzilla is using this definition of TestBlocker:
> 	A test blocker is a bug that prevents at least one test (test case) from
> being executed.
> 
> As far as I can tell this issue does not block anything. The tests execute and
> correctly detect a bug.
> 
> It would be a TestBlocker if it was e.g. a bug in installer which prevented
> the install and thus prevented some other test cases from being executed.
> 
> AFAIK this is not the case. Or am I wrong?
> 
>>> I fit fails for such a long time, we should really invest some time to
>>> look for the root cause of failure(s). If the appointed person does not
>>> have time for this, he/she should be able to negotiate some time
>>> allocated for the investigation. If you feel that the test failures are
>>> not getting enough attention from us then you perhaps need to be more
>>> proactive in the reporting.
>>
>> I am quite OK if Peter Spacek receives some more time for investigation
>> of the root cause of the problem. What I am not OK with is having a
>> perfectly functional testsuite for otherwise working feature, that is
>> being deferred for months just because we do not approve of issue
>> workarounds.
> 
> Sorry, I do not understand this. What do you mean?

What I mean is:
Do we support adding dnssec-enabled zones?
  - Yes
Does a dnssec-enabled zone display signature?
  - Yes, but after restarting of named-pkcs11.
Should this feature in it's current state be covered with automated tests
  - YES, definitely!
But I understand that most of the team have the opposite opinion on the
subject

We could probably add a separate test that checks exactly this bug: add
a dnssec-enabled zone and query it WITHOUT restarting named-pkcs11. And
mark it as xfail. This way we hit 2 goals simultaneously: test the
feature itself and have a constant reminder for you guys, that we still
have this problem that needs your attention.

> 
> 
>>> We really should be fixing issues, not adding workarounds so that tests
>>> pass.
> 
> +1, it is just a matter of priority
> 

-- 
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.