[PATCH] ci: Install python RPMs from libvirt/libvirt-python CI artifacts

[Daniel P. Berrangé]

On Thu, Oct 13, 2022 at 09:43:50AM +0200, Erik Skultety wrote:
> On Mon, Oct 10, 2022 at 11:09:51AM +0100, Daniel P. Berrangé wrote:
> > On Mon, Oct 10, 2022 at 11:46:43AM +0200, Peter Krempa wrote:
> > > After addition of the new libvirt-client-qemu sub-package which is using
> > > python bindings (thus creating a circular dependency between the libvirt
> > > and libvirt-python projects) the integration jobs fail with:
> > > 
> > >   Error:
> > >    Problem: conflicting requests
> > >     - nothing provides python3-libvirt >= 8.9.0-1.el9 needed by libvirt-client-qemu-8.9.0-1.el9.x86_64
> > 
> > Hmm, I wonder, should we really be trying to install *all* RPMs ?
> > 
> > IIUC, we're only running integration tests for QEMU, so it ought
> > to be sufficient to do only have the QEMU driver and virsh client
> > and Perl APIs. eg can we change ci/integrate-templates.yml to
> > do
> > 
> >    createrepo libvirt-rpms
> >    createrepo libvirt-perl
> >    dnf install libvirt-daemon-qemu libvirt-client perl-Sys-Virt
> 
> Sorry for the delay, I was actually integrating ^this suggestion and GitLab is,
> well, slow to test these changes with. Anyhow, I vote for Peter's original
> patch as it's the simplest and cleanest way to fix the problem, why? Here's
> what I ended up with on my branch:
> 
> diff --git a/ci/integration-template.yml b/ci/integration-template.yml
> index d4799b2ca8..9c58a1b71c 100644
> --- a/ci/integration-template.yml
> +++ b/ci/integration-template.yml
> @@ -25,7 +25,31 @@
> 
> 
>  .install-deps: &install-deps
> -  - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
> +  - sudo dnf install -y createrepo
> +  - createrepo libvirt-rpms
> +  - createrepo libvirt-perl-rpms
> +  - |-
> +    for repo in libvirt-rpms libvirt-perl-rpms; do
> +    cat << EOF >> "$SCRATCH_DIR/ci.repo"
> +    [CI-$repo]
> +    name=ci-$repo
> +    baseurl=file://$(pwd)/$repo
> +    enabled=1
> +    gpgcheck=0
> +    EOF
> +    done
> +  - sudo mv "$SCRATCH_DIR/ci.repo" /etc/yum.repos.d/
> +  - sudo dnf install -y
> +          libvirt-daemon-config-nwfilter
> +          libvirt-daemon-config-network
> +          libvirt-daemon-driver-interface
> +          libvirt-daemon-driver-network
> +          libvirt-daemon-driver-nodedev
> +          libvirt-daemon-driver-nwfilter
> +          libvirt-daemon-driver-qemu
> +          libvirt-daemon-driver-secret
> +          libvirt-daemon-driver-storage
> +          perl-Sys-Virt
>    - sudo pip3 install --prefix=/usr avocado-framework

FYI, can be simplified to:

  - sudo dnf install -y createrepo
  - createrepo libvirt-rpms
  - createrepo libvirt-perl-rpms
  - dnf config-manager --add-repo libvirt-rpms
  - dnf config-manager --add-repo libvirt-perl-rpms
  - sudo dnf install -y
          libvirt-daemon-config-nwfilter
          libvirt-daemon-config-network
          libvirt-daemon-qemu
          perl-Sys-Virt

> Not only does it require more steps, is ugly because of the heredoc combined
> with YAML's explicit line break preservation (|-), libvirt-daemon-driver-qemu
> also doesn't pull all the needed bits as it was suggested, e.g.
> libvirt-daemon-config-nwfilter which is needed by the tests. Although the tests
> don't need virtinterfaced, I still had to include it because the logic which
> restarts all the daemons would have to be adjusted as well; and then one day we
> add a test for virtinterfaced we'd have to add it back, so IMO installing and
> restarting all the daemons is the cleanest and most transparent way of setting
> up the CI env even though we're not making use of all of it now (and so I ended
> up naming all the secondary drivers explicitly).

NB, I suggested libvirt-daemon-qemu, not libvirt-daemon-driver-qemu,
because the former pulls in all the secondary driver packages, so
would only miss the config packages.

> The other huge inconvenience with the proposed approach compared to Peter's is
> debugging the failures. What I currently do with any integration CI failures is
> that I connect to the runner machine and spin up an instance of one of the
> images from which gitlab provisions the environment, download the RPM artifacts
> from libvirt and libvirt-perl, install and then debug - again, why? Because
> unfortunately trying to do this locally is still less convenient even though
> we've done plenty of work in lcitool to improve the status quo. So, back to
> your suggestion - I'd have to create the repositories manually, install a few
> explicit packages and then test; sorry, that maybe fine for the CI, but doesn't
> beat the following in any way:
> 
> wget <libvirt-rpms-uri> -O libvirt_rpms.zip
> wget <libvirt-perl-rpms-uri> -O libvirt_perl_rpms.zip
> wget <libvirt-python-rpms-uri> -O libvirt_python_rpms.zip
> unzip <yada yada>
> 
> dnf install libvirt_rpms/* libvirt_perl_rpms/* libvirt_python_rpms/*

That's fine as an approach for reproducing locally. Does it matter
that you're installing more packages that the CI system would have ?
I guess it could affect the results, but fairly slim liklihood IMHO.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|