[Avocado-devel] Redesign Avocado-VT with common abstraction of all the tests

[Satheesh Rajendran]

On Tue, May 28, 2019 at 04:17:42PM +0530, Balamuruhan S wrote:
> Hi All,
> 
> This is a proposal to abstract all the setups, test run,
> post test validation, cleanup of the tests from tp-libvirt/tp-qemu
> into avocado-vt so that we have one unique testcase source code
> and existing test config can grow based on the new scenarios/variants.

Hi Bala and all,

This would be a very good one for adding/testing newer/complex test 
scenarios with very less/no code to add in future.

Here are my suggestions/proposals to make this a reality and helpful across.

To get full benefit from this proposal, we need to have few initial 
changes in our framework, some of which are having RFC in community 
and some of them we need to work towards it continually.

[1]. Re-order of test configs, so that we elimiate test specific params 
and will help us achive overall modularity,
RFC patch: https://github.com/avocado-framework/avocado-vt/pull/1753 [a]
Testcase RFC PR, https://github.com/autotest/tp-libvirt/pull/1829 [b]

To achieve this all of our testcases[b] needs a change to use the 
global/common vm params and once that is done, we need to apply 
the PR [a] in order to avoid any regressions.

Why should we do this change, checkout the commit message of [b] for details.

[2]. We need to add support for all relevant parameters that virt-install 
commandline supports,

It can either done in libvirt_vm.py through methods like add_numa(),add_smp()
(or)
User can do that easily through --vt-extra-params as well,
https://github.com/avocado-framework/avocado-vt/commit/bd54bf531ad57e7e7f83446a7986fb41d7b083b1

Idea is to make use of create_vm_libvirt and avoid any additional vm destroy/start 
inside testcase just to change vm configs and save runtime of tests,
all the needed vm configrations is set at the time of vm import in case tp-libvirt, 
whereas this is already the case in tp-qemu.
Background: https://www.redhat.com/archives/avocado-devel/2018-May/msg00011.html

[3]. We need to have more and more vm events and host events and validations 
to be coded in library rather than testcase which would help us in old tests 
and for this proposal aswell.

vm events: hotplug_vcpu()
           hotplug_mem()
           hotplug_disk()...
host events: host_offline_mem()
             host_set_hugepage()
             host_cpu_idle() ....

Similarly, for validating the events, we should have library method so that
we can add future places to check there itself.
For example, to check whether vcpu is hotplugged or not
we need to verify in many places like vcpucount, vcpuinfo, xml, inside guest vm, 
recently I had to add check from domstats output and verify numa aswell.
More than 4 testcase files using it, i just had to change in one place in library as below.
check_vcpu_value()
https://github.com/avocado-framework/avocado-vt/blob/master/virttest/utils_hotplug.py#L469

[4]. In order not too much deviate from framework, we can make use of preprocess 
and postprocess from env_process.py, so that we need not complicate the vm creation 
and pre-requisite needed in case of hugepage configuration and other host setups 
needed for certain tests during vm creation stage.
We can improve and modularize the preprocess and postprocess in env_process.py.

We use this in some of our testcases already, but this has to be mandated across,
like setup_migration(), setup_hugepage(), setup_sriov() etc..., 
this can easily be configured through params like setup_migration=yes,hugepage=yes etc..

---

As Balamuruhan pointed, through this exercise 
we can make avocado-vt more refined, modular, robust and ease of use for testing.

Above said all the changes would be helpful for our already available tests,
So that we are not re-inventing the wheel and which would be stepping stone 
for Balamuruhan's proposal.

Once we(maintainers) are all on same page on how to/whether to go about this proposal, 
if we can help contributors review and suggest inline with discussion here, 
will ease the newer additions.

Request all maintainers to comment on the Balamuruhan and mine suggestions/proposals 
so that we can work together towards achieving it.

Thanks in advance!

Regards,
Satheesh.
> 
> >From testconfig we could pass following params,
> 
> `test_class` - Actual functionality test
> `pre_events` - integrate any other events or test before actual test
> `post_events`- integrate any other events or test after actual test
> 
> Test case file source can be like below common for all the tests,
> 
> def run(test, params, env):
+1
>     """
>     Test KVM scenarios
>     """
>     try:
>         test_class = eval(params.get("test_class"))
>         pre_events = params.get("test_pre_events").split()
>         post_events = params.get("test_post_events").split()
>         vms = env.get_all_vms()
>         try:
>             for each in pre_events:
>                 preevent = eval(each)
>                 preevent_obj = preevent(vms, params)
> 		preevent_obj.run()
>             test_obj = test_class(vms, params)
>             test_obj.run()
>             for each in post_events:
>                 postevent = eval(each)
>                 postevent_obj = postevent(vms, params)
>                 postevent_obj.run()
>         except Exception as info:
>             test.fail(info)
>         finally:
>             if preevent_obj:
>                 preevent_obj.cleanup()
>             if test_obj:
>                 test_obj.cleanup()
>             if postevent:
>                 postevent_obj.cleanup()
>     except NameError as info:
>         test.error(info)
> 
> Avocado-VT should have class for each functionality test or events 
> 
> class TestClass(object):
we can reuse this part as in env_process.py
>     def __init__(self, vms, params):
>         self.vms = vms
> 	self.params = params
> 
>     # perform pre test setup
>     def setup(self):
>     def run(self):
>         self.setup()
>         # code that performs actual test
> 	self.validation
>     # perform post test validation
>     def validation(self):
>     def cleanup():
> 
> As example we can take migration test,
> One of the test performs CPU hotplug before migration, perform migration
> and CPU hotunplug after migration
> 
> * preevent will be CPU hotplug Test
> * test_class will be Migration Test
> * postevent will be CPU hotunplug Test
> 
> We can control the preevent or postevent from params,
> 
> 1. require it or not (migration with/without any events)
> 2. require same no of multiple events before/after test (hotplug event
>    before migration and hotunplug event after migration)
> 3. require variable no of multiple events before/after test (hotplug
>    event, stress event before and hotunplug event after migration)
> 
> By this design,
> 
> * Single testcase source for all the functional tests.
> * All the functional code abstracted to Avocado-VT into its own class
>   for each features respectively.
> * Avoids all the duplicate named params for every test introduced and
>   similar logics that reduces the code bloat in Autotest and Avocado-VT.
> * Make almost all the attributes configurable from params.
> * New testcases can be generated dynamically from avocado commandline by
>   giving different param values using --vt-extra-params,
> 
>   Example: we can change no of cpus to be hotplugged/hotunplugged while
>            triggering the test from avocado command line instead of writing new
>            testcase for it.
> 
> Initially it would take effort to refactor Avocado-VT to adopt it but,
> I think it will greatly change the way we can test going forward and
> make the framework more simple, powerful and robost.
> 
> Please share your thoughts on it, I can make a working model based on
> the feedback and comments on the proposal.
> 
> -- Bala
>