Libvirt CI for running functional tests

[Erik Skultety]

On Thu, May 27, 2021 at 11:17:04AM -0500, Praveen K Paladugu wrote:
> Hi,
> 
> While developing cloud-hypervisor driver for libvirt, we re-fitted
> cloud-hypervisor project's CI to libvirt. This CI was built on Rust and
> currently supports VM boot up tests.
> 
> https://github.com/cloud-hypervisor/libvirt/tree/ch/ch_integration_tests

Hi,
so excited to hear about ^this effort - I haven't gone over the code base yet,
but nevertheless having something set up already at your side is awesome.

> 
> We are working on extending this CI to incorporate more functional tests:
> Networking, Thread Pinning etc. We are curious to know if libvirt project
> has any plans to setup a CI to run functional tests.
> 
> I noticed https://gitlab.com/libvirt/libvirt-ci effort which focuses on
> running builds against various platforms and formats. Could you please
> clarify libvirt project's plan for setting up a CI to run functional tests.
> 

Yes, we do have plans. As you've already correctly noted, currently we only
have automated builds running in GitLab the configuration of which comes and is
maintained by the libvirt-ci project. As for the functional tests though, we're
currently fighting on 3 fronts:
    - infrastructure
    - automation machinery
    - testing framework

Without going in too much detail we're working on improving libvirt-ci project
in a way that would allow to make libvirt CI configurable for any interested
party, IOW we'd set a standard on what the machines should look like and how
libvirt expects to interact with them \wrt executing tests on them, but what
tests you run on them is up to you as long as you report them back to the
upstream libvirt pipeline (most likely gitlab).

To go to a little more detail:

Infrastructure
--------------
- we're planning on running the workloads in nested KVM VMs, because it's much
  easier to set up in an automated fashion, is suitable to be run on local
  developer's machines and can be thrown away afterwards

- machines will be connected to gitlab with the gitlab-runner agent
 (provided we don't decide to move to a different platform which is not likely
  even with the restriction on using public runners with the free tier)

Automation
----------
- this will be handled by libvirt-ci (lcitool) which already has support for
  both container and VM workloads, we just have to shape it so that it can do
  what we want in terms of functional tests in VM workloads

Test framework
--------------
- there already are 2 frameworks: Avocado-VT and libvirt TCK none of which
  ultimately will be used, because they're not particulary libvirt developer
  friendly for various reasons (not the point of this email)

- we're experimenting with using the plain Avocado framework integration just
  like QEMU has already done upstream (so we'd like to stay in aligned with QEMU)

- the main arguments for selecting the Avocado framework are (there a few more):
    -> the native language of Avocado is Python which we already decided in
       upstream libvirt to be the dominant, modern and preferred scripting
       language of many users/contributors

    -> Avocado is truly language agnostic as far as tests go, so if you're
       bringing an existing test suite in a different language like e.g. Bash,
       that is fine Avocado can detect and execute external tests as well

    -> Avocado supports many of the new test result formats out there along
       with the simple and older TAP format, so it can satisfy various needs

    -> Avocado supports parallel test executions

- as for what the initial deployment will utilize, it's impossible to think
  that we're going to port all of Avocado-VTs 17k test case (hell no!), so we'd
  start with the older test set coming from libvirt TCK which has proven to
  catch serious bugs, but new tests would be written solely in Python and
  very very slowly we'd migrate the TCK test cases from Perl to Python and host
  all of it as part of the main libvirt repo

I hope that ^this gist would be enough of an answer to you :). Stay tuned for
next plans on the CI front.

Regards,
Erik