[Avocado-devel] Tips for making a standalone test script Avocado-friendly?

[Cleber Rosa]

On 04/06/2017 02:20 PM, Eduardo Habkost wrote:
> On Thu, Apr 06, 2017 at 12:38:31PM -0400, Cleber Rosa wrote:
> [...]
>>>>> Do you have any suggestions for making the test result output
>>>>> from those scripts easily consumable by the Avocado test runner?
>>>>>
>>>>
>>>> If those tests are treated as SIMPLE tests, then all of their output
>>>> will automatically be added to the Avocado logs.
>>>>
>>>> Now, I do understand what your hopes are (or were after my possibly
>>>> disappointing recommendations :).  Ideally, Avocado would be able to
>>>> find the tests available on your standalone scripts[6], would know how
>>>> to run the individual tests[7].  Right now, this requires writing a
>>>> plugin.  I can see some ideas for a "Python universal" plugin here, that
>>>> would find other hints of tests in plain Python source code.  If you
>>>> think that makes sense, let's discuss it further.
>>>>
>>> Actually we already have `robot` plugin which is able to discover
>>> `robot`-like tests. How about creating a `unittest` loader, which would
>>> do the `avocado-find-unittest` and discover them individually? By
>>> default nothing would change as the `.py` files would be recognized as
>>> instrumented/simple tests, but when you change the order of loaders it'd
>>> detect them as unittests:
>>>
>>
>> Eduardo mentioned that he'd eventually get rid of the unittest
>> dependency.  That's why I was suggesting/brainstorming about an even
>> more generic way of providing hints that there are tests on a Python
>> source file.
> 
> I planned to get rid of the unittest dependency because it did
> not seem useful to me. But if it provides Avocado test discovery
> ability for free, I would probably keeping it.
> 

OK, so that let's keep track of that:

https://trello.com/c/j9IE7BHy/992-loader-add-native-python-unittest-support

> Now, another problem is that I wish a test granuality that
> requires running QEMU at least once during the test discovery
> phase (e.g. one test case for each machine-type/device/CPU).
> Maybe this will break some assumptions in Avocado?
> 

That would be possible, again, with a custom plugin.  The Avocado-VT
loader, for instance, does "what-not" during test discovery.  How to do
that in a generic (and requirements free) way, may be a little more
tricky (or interesting).

One way of possibly doing this (which is halfway there, but still not
supported), is to use the `avocado.core.Job` API, which we intend to
make public in the future.

You could still have your standlone scripts (added simplicity here if
they're already unittest.TestCase based), and write a custom Job that
would create a custom test suite:

...
# regular script content
...
if __name__ == '__main__':
   if under_avocado(): # fictitious function
      from avocado import Job
         class QemuDynamicJob(Job):
            def create_test_suite(self):      # [1]
               methods = find_tests_on_this_file()
               self.test_suite = methods * get_cpu_models() # [2]
      QemuDynamicJob().run()

> Are tests able to get a few input parameters during thest
> discovery? e.g. I want to tell the test script/module the list of
> QEMU binaries I want to test. Today I use an environment variable
> or command-line parameter for that.
> 

Right now a custom loader can have access to all sorts of parameters.
Tests do not discover themselves, so that would be against the current
architecture, but jobs do, as explained before.

Let me know if that makes sense.

[1] -
http://avocado-framework.readthedocs.io/en/48.0/api/core/avocado.core.html#avocado.core.job.Job.create_test_suite

[2] -
http://avocado-framework.readthedocs.io/en/48.0/api/core/avocado.core.html#avocado.core.job.Job.test_suite

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170407/eb064b59/attachment.sig>