Libvirt CI for running functional tests

[Praveen K Paladugu]

Erik,

Thanks for this detailed response. It answers many questions we had 
about CI for libvirt.


On 6/2/2021 3:32 AM, Erik Skultety wrote:
> On Thu, May 27, 2021 at 11:17:04AM -0500, Praveen K Paladugu wrote:
>> Hi,
>>
>> While developing cloud-hypervisor driver for libvirt, we re-fitted
>> cloud-hypervisor project's CI to libvirt. This CI was built on Rust and
>> currently supports VM boot up tests.
>>
>> https://github.com/cloud-hypervisor/libvirt/tree/ch/ch_integration_tests
> 
> Hi,
> so excited to hear about ^this effort - I haven't gone over the code base yet,
> but nevertheless having something set up already at your side is awesome.
> 
>>
>> We are working on extending this CI to incorporate more functional tests:
>> Networking, Thread Pinning etc. We are curious to know if libvirt project
>> has any plans to setup a CI to run functional tests.
>>
>> I noticed https://gitlab.com/libvirt/libvirt-ci effort which focuses on
>> running builds against various platforms and formats. Could you please
>> clarify libvirt project's plan for setting up a CI to run functional tests.
>>
> 
> Yes, we do have plans. As you've already correctly noted, currently we only
> have automated builds running in GitLab the configuration of which comes and is
> maintained by the libvirt-ci project. As for the functional tests though, we're
> currently fighting on 3 fronts:
>      - infrastructure
>      - automation machinery
>      - testing framework
> 
> Without going in too much detail we're working on improving libvirt-ci project
> in a way that would allow to make libvirt CI configurable for any interested
> party, IOW we'd set a standard on what the machines should look like and how
> libvirt expects to interact with them \wrt executing tests on them, but what
> tests you run on them is up to you as long as you report them back to the
> upstream libvirt pipeline (most likely gitlab).
> 
> To go to a little more detail:
> 
> Infrastructure
> --------------
> - we're planning on running the workloads in nested KVM VMs, because it's much
>    easier to set up in an automated fashion, is suitable to be run on local
>    developer's machines and can be thrown away afterwards
> 
> - machines will be connected to gitlab with the gitlab-runner agent
>   (provided we don't decide to move to a different platform which is not likely
>    even with the restriction on using public runners with the free tier)
> 
> Automation
> ----------
> - this will be handled by libvirt-ci (lcitool) which already has support for
>    both container and VM workloads, we just have to shape it so that it can do
>    what we want in terms of functional tests in VM workloads
> 
> Test framework
> --------------
> - there already are 2 frameworks: Avocado-VT and libvirt TCK none of which
>    ultimately will be used, because they're not particulary libvirt developer
>    friendly for various reasons (not the point of this email)
> 
> - we're experimenting with using the plain Avocado framework integration just
>    like QEMU has already done upstream (so we'd like to stay in aligned with QEMU)
> 
> - the main arguments for selecting the Avocado framework are (there a few more):
>      -> the native language of Avocado is Python which we already decided in
>         upstream libvirt to be the dominant, modern and preferred scripting
>         language of many users/contributors
> 
>      -> Avocado is truly language agnostic as far as tests go, so if you're
>         bringing an existing test suite in a different language like e.g. Bash,
>         that is fine Avocado can detect and execute external tests as well
> 
>      -> Avocado supports many of the new test result formats out there along
>         with the simple and older TAP format, so it can satisfy various needs
> 
>      -> Avocado supports parallel test executions
> 
> - as for what the initial deployment will utilize, it's impossible to think
>    that we're going to port all of Avocado-VTs 17k test case (hell no!), so we'd
>    start with the older test set coming from libvirt TCK which has proven to

What do you mean by "older test set" here? Could you please point me to 
some links/docs related to this test set?

If this test set has sufficient coverage for our usecase, we will 
consider pivoting to avocado suite for future test cases.

>    catch serious bugs, but new tests would be written solely in Python and
>    very very slowly we'd migrate the TCK test cases from Perl to Python and host
>    all of it as part of the main libvirt repo
> 
> I hope that ^this gist would be enough of an answer to you :). Stay tuned for
> next plans on the CI front.
> 
> Regards,
> Erik
> 

Lastly, where can we follow the adoption of Avocado Framework? Will this 
start in libvirt-ci repository?


-- 
Regards,
Praveen K Paladugu