[Avocado-devel] [avocado-devel] Negative testcases

[Lukáš Doktor]

Hello guys,

exactly, sorry for late response, we are a bit busy before the DevConf.

So to wrap it up I'm not aware of any existing generic test, but there 
are specific tests which create the machine in the test's body, so you 
can inspire by them. One of them is `virtio_console` but basically you 
can just `fgrep "env_process" .` to see them all.

About the implementation, avocado uses cartesian config params to decide 
which machines and what types will be created before the test execution 
and cleans them up after the execution. All of this can be overridden so 
you can say `start_vm = no` to ignore the pre-process of all machines, 
or `start_vm_$name = no` to ignore just one specific vm. Than you can 
change the params and re-run the pre-process of that particular VM.

A rough example from our session:

```diff
diff --git a/generic/tests/boot.py b/generic/tests/boot.py
index bf4ddc9..19f79ec 100644
--- a/generic/tests/boot.py
+++ b/generic/tests/boot.py
@@ -3,7 +3,7 @@ import logging

  from autotest.client.shared import error

-from virttest import utils_test
+from virttest import utils_test, env_process


  @error.context_aware
@@ -21,15 +21,22 @@ def run(test, params, env):
      """

      timeout = float(params.get("login_timeout", 240))
+    print "xxxxxxxxxxxxxxxxxxxxxxxxxxxx The machine is not yet started, 
is it?"
+    params["start_vm"] = "yes"
+    env_process.preprocess_vm(test, params, env,
+                              params["main_vm"])
      vms = env.get_all_vms()
      for vm in vms:
```

When the `start_vm` is `no`, this starts the machine inside the test. 
With `start_vm == yes` it starts the vm before test execution and 
re-checks whether it should be re-created inside the test. For example 
if you use `params["mem"] = 123456` it detects that the running VM 
differs from the newly defined one, shuts the previous one off and 
recreates it with 123456 MB of ram.

Anyway the whole workflow of avocado-vt test execution is:

1. parse cartesian config
2. run preprocess in env_process
3. run test
4. run postprocess in env_process
5. report results

but you can always call the methods earlier or multiple times.

Kind regards,
Lukáš

PS: Sorry about the format, it's a summary of our real-life meeting and 
hopefully wraps all what was said there.

Dne 25.1.2017 v 16:33 Radek Duda napsal(a):
> I spoke to Lukas and Cleber personally today so it is clear now. The
> thing is that I have to put off preprocess_vm function by setting
> start_vm = no. and put it into try..except block.
>
> Radek
>
> On Wed, Jan 25, 2017 at 1:37 PM, Lucas Meneghel Rodrigues
> <lookkas at gmail.com <mailto:lookkas at gmail.com>> wrote:
>
>     Isn't catching the exception and checking if its fields look like
>     what you expect an option? That is the sort of thing that in my mind
>     wouldn't need additional support.
>
>     On Tue, Jan 24, 2017 at 4:33 PM Radek Duda <rduda at redhat.com
>     <mailto:rduda at redhat.com>> wrote:
>
>         Hi,
>
>         I wonder if there is any possiblity to perform negative
>         testcases in avocado / avocado-vt?
>
>         To be more specific:
>         If I supply some nonsence to SPICE listening port as qemu-kvm
>         parameter, the creation of VM should fail. Indeed it fails -
>         throws VMCreateError, but that is all.
>
>         Is there any way how to mark this TC as passed?
>
>         Thanks for reply.
>
>
>         Regards,
>         Radek Duda
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 502 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170126/51a2c8c3/attachment.sig>