[Freeipa-devel] [TEST][patch-0037]Fixes of dnssec tests

[Martin Basti]

On 06.05.2016 16:38, Oleg Fayans wrote:
>
> On 05/06/2016 03:25 PM, Petr Spacek wrote:
>> On 6.5.2016 15:03, Oleg Fayans wrote:
>>>
>>> On 05/06/2016 12:08 PM, Martin Babinsky wrote:
>>>> On 05/06/2016 11:14 AM, Oleg Fayans wrote:
>>>>>
>>>>> On 05/06/2016 09:48 AM, Martin Basti wrote:
>>>>>>
>>>>>> On 06.05.2016 09:36, Oleg Fayans wrote:
>>>>>>> Tests are finally stable:
>>>>>>>
>>>>>>> ============================= test session starts
>>>>>>> ==============================
>>>>>>> platform linux2 -- Python 2.7.11 -- py-1.4.30 -- pytest-2.7.3
>>>>>>> rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile: pytest.ini
>>>>>>> plugins: multihost, sourceorder
>>>>>>> collected 8 items
>>>>>>>
>>>>>>> test_integration/test_dnssec.py ........
>>>>>>>
>>>>>>> ========================= 8 passed in 5561.48 seconds
>>>>>>> ==========================
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> PATCH 38 LGTM
>>>>>>
>>>>>> PATCH 37 IIRC I refused to accept workaround for this issue when you
>>>>>> send this (almost the same) patch for first time, are you sure that we
>>>>>> want to hide real issues in tests, to just have green color there?
>>>>>>
>>>>> The underlying issue is 7 months old. Latest update in the issue from
>>>>> Peter Spacek is: "I do not have time to investigate this issue now",
>>>>> which means, that it will stay there for unpredictable amount of time
>>>>> more. If we want to have a "green" jenkins that actually tests existing
>>>>> features, we have to accept workarounds for such long-term issues
>>>>>
>>>>>> Martin
>>>> I have never been a big fan of "having a green jenkins whatever it
>>>> takes" but I understand that there are all kinds of pressure on your
>>>> team to deliver 100% stable test results.
>>>>
>>>> If the test fails, let it fail or, even better, use 'xfail' markers so
>>>> that we know that this test fails and we should investigate.
>>> Then all 8 existing cases would have to be marked as xfailed.
>> That is perfectly fine - the test simply found a bug and we have to fix it.
>> There is no point in having "green" tests just to have them.
>>
>> Let me clarify my comment in the ticket that this is not a test blocker:
>> Red Hat Bugzilla is using this definition of TestBlocker:
>> 	A test blocker is a bug that prevents at least one test (test case) from
>> being executed.
>>
>> As far as I can tell this issue does not block anything. The tests execute and
>> correctly detect a bug.
>>
>> It would be a TestBlocker if it was e.g. a bug in installer which prevented
>> the install and thus prevented some other test cases from being executed.
>>
>> AFAIK this is not the case. Or am I wrong?
>>
>>>> I fit fails for such a long time, we should really invest some time to
>>>> look for the root cause of failure(s). If the appointed person does not
>>>> have time for this, he/she should be able to negotiate some time
>>>> allocated for the investigation. If you feel that the test failures are
>>>> not getting enough attention from us then you perhaps need to be more
>>>> proactive in the reporting.
>>> I am quite OK if Peter Spacek receives some more time for investigation
>>> of the root cause of the problem. What I am not OK with is having a
>>> perfectly functional testsuite for otherwise working feature, that is
>>> being deferred for months just because we do not approve of issue
>>> workarounds.
>> Sorry, I do not understand this. What do you mean?
> What I mean is:
> Do we support adding dnssec-enabled zones?
>    - Yes
> Does a dnssec-enabled zone display signature?
>    - Yes, but after restarting of named-pkcs11.
> Should this feature in it's current state be covered with automated tests
>    - YES, definitely!
> But I understand that most of the team have the opposite opinion on the
> subject
>
> We could probably add a separate test that checks exactly this bug: add
> a dnssec-enabled zone and query it WITHOUT restarting named-pkcs11. And
> mark it as xfail. This way we hit 2 goals simultaneously: test the
> feature itself and have a constant reminder for you guys, that we still
> have this problem that needs your attention.
>

Yes why not, we can duplicate code for the every issue.
We will have nice statics on github :)

>>
>>>> We really should be fixing issues, not adding workarounds so that tests
>>>> pass.
>> +1, it is just a matter of priority
>>