[Freeipa-devel] [PATCH 0081] Add initial tests for OTP

[Martin Kosek]

On 01/28/2015 11:58 PM, Nathaniel McCallum wrote:
> Attached is a new version which fixes most of the issues found.
> 
> Comments below.

Thanks!

> 
> On Mon, 2015-01-12 at 15:53 +0100, Petr Viktorin wrote:
>> On 01/06/2015 03:26 AM, Nathaniel McCallum wrote:
>>> On Thu, 2014-11-20 at 11:13 -0500, Nathaniel McCallum wrote:
>>>>> This tests the general workflow for OTP including most possible
>>>>> token combinations. This includes 5872 tests. Further optimization
>>>>> is possible to reduce the number of duplicate tests run.
>>>>>
>>>>> Things not yet tested:
>>>>> * ipa-kdb
>>>>> * ipa-otpd
>>>>> * otptoken-sync
>>>>> * RADIUS proxy
>>>>> * token self-management
>>>>> * type specific attributes
>>> Attached is the latest iteration of the OTP test work. This now includes
>>> all major cases except token self-management and RADIUS proxy (which
>>> will come in its own patch). Token self-management is held up by the
>>> fact that I can't get alternate ccaches to work with the API. I have
>>> tried kinit-ing to an independent ccache and exporting KRB5CCNAME, but
>>> this doesn't work for some reason I can't figure out.
>>>
>>> I ended up creating my own fixture mechanism. I'm not in love with it,
>>> but it is simple and at least gets the scoping correct. It also
>>> generates individual tests for each parameterized state, so the output
>>> is both correct and obvious.
>>>
>>> I also implemented OTP myself. This isn't much code, but pyotp has a
>>> major bug and is dead upstream. I'd like to migrate to
>>> python-cryptography when it lands as a dependency of FreeIPA. But due to
>>> timing issues, we can't land it now. This will be a small patch in the
>>> future.
>>>
>>> Even with the caveats above, I feel like the test coverage provided by
>>> this test is worth review/merge. As a rough estimate, I think this is
>>> about 70% code coverage. Of the remaining coverage, I see:
>>> * RADIUS proxy - 10%
>>> * token self-management - 10%
>>> * misc testable - 5%
>>> * misc untestable - 5%
>>>
>>> All tests in this patch succeed on 4.1.2.
>>
>> It doesn't work for me on the 4.2 branch, see the attached errors. Not 
>> sure what I'm doing wrong.
> 
> This doesn't work at all on the 4.2 branch. This is because nose, like
> the upstream test harnesses, executes yielded tests as they are yielded.
> However, pytest does not. It collects all the yielded tests into a
> single bundle and then runs them after the state of their fixtures is
> already destroyed. This complete breaks the fixturize() decorator.
> 
> Why are we using pytest again? It may offer nifty features, but it sucks
> at integration testing.

While right now, I am mostly concerned about OTP testing in FreeIPA 4.1.* as
this is the release going in downstream releases, the next step after
successfully reviewing&testing of ipa-4-1 incarnation of the packages will be
update in master branch, i.e. making them work with our favorite testing
framework - pytest :-).

Martin