[almighty] Database test patterns

[Konrad Kleine]

Hi Thomas,

I ran into this problem a while back and though I though I had it fixed at
least for my work item type black box tests [1].

I've been using the "suite" package [2] from the testify repository. You
have to move your TestXXX functions to methods of a struct with additional
special functions (your suite struct). These allow you to guarantee certain
preconditions are met before and after any of the tests is run (SetupSuite,
and TearDownSuite). If you need special preconditions to be fulfilled
before or after an individual test is run, you can also ensure that with
two methods (SetupTest, TearDownTest).

I'm pretty sure that those special methods will be executed even if a test
fails.

All in all, this should elegantly solve the problem of having a "clean"
database before and after a test is run, right? In my test [1] I blindly
remove the DB entries, that I want to create in the test, before and after
a test is run.

Regards,
Konrad

[1] Work item type black box tests:
https://github.com/almighty/almighty-core/blob/fd204a0420e4a454832b6867f2b556d244f7f646/workitemtype_blackbox_test.go#L97
[2] Suite Package: https://github.com/stretchr/testify#suite-package

On Thu, Oct 27, 2016 at 8:28 AM, Dhriti Shikhar <dshikhar at redhat.com> wrote:

> On Tuesday 25 October 2016 09:09 PM, Thomas Mäder wrote:
>
> Hi folks,
>>
>> in PR https://github.com/almighty/almighty-core/pull/294 I have used the
>> pattern to roll back to execute tests in a transaction which is rolled back
>> at the end of the test. Thus, the test is guaranteed to not change the
>> database. There were couple of objections to this patterns, which leads me
>> to seek some convergence on the topic of db tests.
>>
> This approach works well and it is quite effective in isolating changes
> between tests. Except in the case where you need to test the transaction
> itself.
>
>
>
>> Every test has certain preconditions. For example, when I test creation
>> of an object with a given ID, the precondition is that this ID does not yet
>> exist in the DB. If the preconditions are not met, the test cannot properly
>> run and thus does not yield any information about the code it is supposed
>> to test.
>>
>> To solve this problem, there are two approaches. The first one is to
>> program the test without having preconditions. For example, instead of
>> creating an object with the ID "42" in the above case, the test could try
>> to use a UUID as the ID for the object it creates. Since there will be no
>> collision with previous runs, there is no precondition that can be violated.
>>
>> The second one is get the system into a well know state before the test
>> runs. One of the patterns to do this is to always start with an empty
>> database. Alternatively, the test setup could contain code to establish the
>> precondition. In our example, the setup would delete the object with the ID
>> "42" (if it exists) in order to make sure that the create call can be
>> tested.
>>
>> One of the problems we have is that we run unit often against a local
>> database while developing. Even if the tests are written to work
>> repeatably, we don't want the test database to be trashed every time nor
>> filled up with useless test data. So ideally, tests should clean up after
>> themselves.
>>
>> The rollback pattern keeps the db from filling up, but can't prevent
>> failed preconditions. Another pattern is to setup objects used in the test
>> before the test and to remove them after the test. However, this doesn't
>> really work either: tear down may fail and leave objects in the database.
>> The next run of the test may then fail.
>>
>> I think the only reliable way to write db tests is this:
>>
>> setup
>>
>>     delete all objects the test expects NOT to exist
>>
>>     delete all objects the test expects to exist (because they may not be
>> in the correct state)
>>
>>     create all objects the test expects to exist.
>>
>> Test
>>
>>     use the objects from above
>>
>> tearDown
>>
>>     remove all objects created in setup or the test
>>
>> What are your proposals on how to structure db tests?
>>
>> /Thomas
>>
>> _______________________________________________
>> almighty-public mailing list
>> almighty-public at redhat.com
>> https://www.redhat.com/mailman/listinfo/almighty-public
>>
>
> _______________________________________________
> almighty-public mailing list
> almighty-public at redhat.com
> https://www.redhat.com/mailman/listinfo/almighty-public
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/almighty-public/attachments/20161027/5a2f4395/attachment.htm>