Libvirt CI for running functional tests

Mon Jun 7 05:51:56 UTC 2021

On Fri, Jun 04, 2021 at 09:52:52AM -0500, Praveen K Paladugu wrote:
> Erik,
> 
> Thanks for this detailed response. It answers many questions we had about CI
> for libvirt.
> 
> 
> On 6/2/2021 3:32 AM, Erik Skultety wrote:
> > On Thu, May 27, 2021 at 11:17:04AM -0500, Praveen K Paladugu wrote:
> > > Hi,
> > > 
> > > While developing cloud-hypervisor driver for libvirt, we re-fitted
> > > cloud-hypervisor project's CI to libvirt. This CI was built on Rust and
> > > currently supports VM boot up tests.
> > > 
> > > https://github.com/cloud-hypervisor/libvirt/tree/ch/ch_integration_tests
> > 
> > Hi,
> > so excited to hear about ^this effort - I haven't gone over the code base yet,
> > but nevertheless having something set up already at your side is awesome.
> > 
> > > 
> > > We are working on extending this CI to incorporate more functional tests:
> > > Networking, Thread Pinning etc. We are curious to know if libvirt project
> > > has any plans to setup a CI to run functional tests.
> > > 
> > > I noticed https://gitlab.com/libvirt/libvirt-ci effort which focuses on
> > > running builds against various platforms and formats. Could you please
> > > clarify libvirt project's plan for setting up a CI to run functional tests.
> > > 
> > 
> > Yes, we do have plans. As you've already correctly noted, currently we only
> > have automated builds running in GitLab the configuration of which comes and is
> > maintained by the libvirt-ci project. As for the functional tests though, we're
> > currently fighting on 3 fronts:
> >      - infrastructure
> >      - automation machinery
> >      - testing framework
> > 
> > Without going in too much detail we're working on improving libvirt-ci project
> > in a way that would allow to make libvirt CI configurable for any interested
> > party, IOW we'd set a standard on what the machines should look like and how
> > libvirt expects to interact with them \wrt executing tests on them, but what
> > tests you run on them is up to you as long as you report them back to the
> > upstream libvirt pipeline (most likely gitlab).
> > 
> > To go to a little more detail:
> > 
> > Infrastructure
> > --------------
> > - we're planning on running the workloads in nested KVM VMs, because it's much
> >    easier to set up in an automated fashion, is suitable to be run on local
> >    developer's machines and can be thrown away afterwards
> > 
> > - machines will be connected to gitlab with the gitlab-runner agent
> >   (provided we don't decide to move to a different platform which is not likely
> >    even with the restriction on using public runners with the free tier)
> > 
> > Automation
> > ----------
> > - this will be handled by libvirt-ci (lcitool) which already has support for
> >    both container and VM workloads, we just have to shape it so that it can do
> >    what we want in terms of functional tests in VM workloads
> > 
> > Test framework
> > --------------
> > - there already are 2 frameworks: Avocado-VT and libvirt TCK none of which
> >    ultimately will be used, because they're not particulary libvirt developer
> >    friendly for various reasons (not the point of this email)
> > 
> > - we're experimenting with using the plain Avocado framework integration just
> >    like QEMU has already done upstream (so we'd like to stay in aligned with QEMU)
> > 
> > - the main arguments for selecting the Avocado framework are (there a few more):
> >      -> the native language of Avocado is Python which we already decided in
> >         upstream libvirt to be the dominant, modern and preferred scripting
> >         language of many users/contributors
> > 
> >      -> Avocado is truly language agnostic as far as tests go, so if you're
> >         bringing an existing test suite in a different language like e.g. Bash,
> >         that is fine Avocado can detect and execute external tests as well
> > 
> >      -> Avocado supports many of the new test result formats out there along
> >         with the simple and older TAP format, so it can satisfy various needs
> > 
> >      -> Avocado supports parallel test executions
> > 
> > - as for what the initial deployment will utilize, it's impossible to think
> >    that we're going to port all of Avocado-VTs 17k test case (hell no!), so we'd
> >    start with the older test set coming from libvirt TCK which has proven to
> 
> What do you mean by "older test set" here? Could you please point me to some
> links/docs related to this test set?

Hi Praveen,
so, documentation in libvirt-derived projects is kinda "problematic"
(read "basically non-existent") and so the best I can do is to share the link
to the repo:
https://gitlab.com/libvirt/libvirt-tck/ (this is the old perl-based framework,
but FWIW it supports arbitrary languages)

As for Avocado-VT:
https://github.com/avocado-framework/avocado-vt (framework)
https://github.com/autotest/tp-libvirt (17k test cases for avocado-vt)

> 
> If this test set has sufficient coverage for our usecase, we will consider
> pivoting to avocado suite for future test cases.
> 
> >    catch serious bugs, but new tests would be written solely in Python and
> >    very very slowly we'd migrate the TCK test cases from Perl to Python and host
> >    all of it as part of the main libvirt repo
> > 
> > I hope that ^this gist would be enough of an answer to you :). Stay tuned for
> > next plans on the CI front.
> > 
> > Regards,
> > Erik
> > 
> 
> Lastly, where can we follow the adoption of Avocado Framework? Will this
> start in libvirt-ci repository?

This will be proposed as an RFC to libvir-list. We intend to make it part of
the main libvirt repo - there are a couple of reasons to do it, but the main
one is to remain aligned with QEMU with our efforts.

Regards,
Erik