[PATCH] ci: Install python RPMs from libvirt/libvirt-python CI artifacts

Thu Oct 13 09:42:06 UTC 2022

On Thu, Oct 13, 2022 at 09:06:53AM +0100, Daniel P. Berrangé wrote:
> On Thu, Oct 13, 2022 at 09:43:50AM +0200, Erik Skultety wrote:
> > On Mon, Oct 10, 2022 at 11:09:51AM +0100, Daniel P. Berrangé wrote:
> > > On Mon, Oct 10, 2022 at 11:46:43AM +0200, Peter Krempa wrote:
> > > > After addition of the new libvirt-client-qemu sub-package which is using
> > > > python bindings (thus creating a circular dependency between the libvirt
> > > > and libvirt-python projects) the integration jobs fail with:
> > > > 
> > > >   Error:
> > > >    Problem: conflicting requests
> > > >     - nothing provides python3-libvirt >= 8.9.0-1.el9 needed by libvirt-client-qemu-8.9.0-1.el9.x86_64
> > > 
> > > Hmm, I wonder, should we really be trying to install *all* RPMs ?
> > > 
> > > IIUC, we're only running integration tests for QEMU, so it ought
> > > to be sufficient to do only have the QEMU driver and virsh client
> > > and Perl APIs. eg can we change ci/integrate-templates.yml to
> > > do
> > > 
> > >    createrepo libvirt-rpms
> > >    createrepo libvirt-perl
> > >    dnf install libvirt-daemon-qemu libvirt-client perl-Sys-Virt
> > 
> > Sorry for the delay, I was actually integrating ^this suggestion and GitLab is,
> > well, slow to test these changes with. Anyhow, I vote for Peter's original
> > patch as it's the simplest and cleanest way to fix the problem, why? Here's
> > what I ended up with on my branch:
> > 
> > diff --git a/ci/integration-template.yml b/ci/integration-template.yml
> > index d4799b2ca8..9c58a1b71c 100644
> > --- a/ci/integration-template.yml
> > +++ b/ci/integration-template.yml
> > @@ -25,7 +25,31 @@
> > 
> > 
> >  .install-deps: &install-deps
> > -  - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
> > +  - sudo dnf install -y createrepo
> > +  - createrepo libvirt-rpms
> > +  - createrepo libvirt-perl-rpms
> > +  - |-
> > +    for repo in libvirt-rpms libvirt-perl-rpms; do
> > +    cat << EOF >> "$SCRATCH_DIR/ci.repo"
> > +    [CI-$repo]
> > +    name=ci-$repo
> > +    baseurl=file://$(pwd)/$repo
> > +    enabled=1
> > +    gpgcheck=0
> > +    EOF
> > +    done
> > +  - sudo mv "$SCRATCH_DIR/ci.repo" /etc/yum.repos.d/
> > +  - sudo dnf install -y
> > +          libvirt-daemon-config-nwfilter
> > +          libvirt-daemon-config-network
> > +          libvirt-daemon-driver-interface
> > +          libvirt-daemon-driver-network
> > +          libvirt-daemon-driver-nodedev
> > +          libvirt-daemon-driver-nwfilter
> > +          libvirt-daemon-driver-qemu
> > +          libvirt-daemon-driver-secret
> > +          libvirt-daemon-driver-storage
> > +          perl-Sys-Virt
> >    - sudo pip3 install --prefix=/usr avocado-framework
> 
> FYI, can be simplified to:
> 
>   - sudo dnf install -y createrepo
>   - createrepo libvirt-rpms
>   - createrepo libvirt-perl-rpms
>   - dnf config-manager --add-repo libvirt-rpms
>   - dnf config-manager --add-repo libvirt-perl-rpms

Oh, I didn't know DNF could do this^^^, that's neat. I still prefer the
simplicity of Peter's patch though.

>   - sudo dnf install -y
>           libvirt-daemon-config-nwfilter
>           libvirt-daemon-config-network
>           libvirt-daemon-qemu
>           perl-Sys-Virt
> 
> > Not only does it require more steps, is ugly because of the heredoc combined
> > with YAML's explicit line break preservation (|-), libvirt-daemon-driver-qemu
> > also doesn't pull all the needed bits as it was suggested, e.g.
> > libvirt-daemon-config-nwfilter which is needed by the tests. Although the tests
> > don't need virtinterfaced, I still had to include it because the logic which
> > restarts all the daemons would have to be adjusted as well; and then one day we
> > add a test for virtinterfaced we'd have to add it back, so IMO installing and
> > restarting all the daemons is the cleanest and most transparent way of setting
> > up the CI env even though we're not making use of all of it now (and so I ended
> > up naming all the secondary drivers explicitly).
> 
> NB, I suggested libvirt-daemon-qemu, not libvirt-daemon-driver-qemu,
> because the former pulls in all the secondary driver packages, so
> would only miss the config packages.

We don't build libvirt-daemon-qemu on CS9, I presume that's a bug.

> 
> > The other huge inconvenience with the proposed approach compared to Peter's is
> > debugging the failures. What I currently do with any integration CI failures is
> > that I connect to the runner machine and spin up an instance of one of the
> > images from which gitlab provisions the environment, download the RPM artifacts
> > from libvirt and libvirt-perl, install and then debug - again, why? Because
> > unfortunately trying to do this locally is still less convenient even though
> > we've done plenty of work in lcitool to improve the status quo. So, back to
> > your suggestion - I'd have to create the repositories manually, install a few
> > explicit packages and then test; sorry, that maybe fine for the CI, but doesn't
> > beat the following in any way:
> > 
> > wget <libvirt-rpms-uri> -O libvirt_rpms.zip
> > wget <libvirt-perl-rpms-uri> -O libvirt_perl_rpms.zip
> > wget <libvirt-python-rpms-uri> -O libvirt_python_rpms.zip
> > unzip <yada yada>
> > 
> > dnf install libvirt_rpms/* libvirt_perl_rpms/* libvirt_python_rpms/*
> 
> That's fine as an approach for reproducing locally. Does it matter
> that you're installing more packages that the CI system would have ?
> I guess it could affect the results, but fairly slim liklihood IMHO.

No, the point I was trying to make is that for ^this use case we still need
Peters libvirt-python MR (which I merged in the meantime btw). The other
argument I have is that we shouldn't deviate in the CI job transcript from what
you would done naturally locally in a manual manner - sure, you can always
copy-paste, but copy pasting

    dnf install libvirt_rpms/* libvirt_perl_rpms/* libvirt_python_rpms/*

will always be faster than copying

    sudo dnf install -y createrepo
    createrepo libvirt-rpms
    createrepo libvirt-perl-rpms
    createrepo libvirt-python-rpms
    dnf config-manager --add-repo libvirt-rpms
    dnf config-manager --add-repo libvirt-perl-rpms
    dnf config-manager --add-repo libvirt-python-rpms
    sudo dnf install -y
           libvirt-daemon-config-nwfilter
           libvirt-daemon-config-network
           libvirt-daemon-qemu
           perl-Sys-Virt

Erik