[libvirt PATCH 0/5] ci: Use GitLab container registry

Daniel P. Berrangé berrange at redhat.com
Mon Jun 1 15:19:46 UTC 2020

On Mon, Jun 01, 2020 at 04:51:19PM +0200, Pavel Hrdina wrote:
> On Fri, May 29, 2020 at 03:00:39PM +0200, Andrea Bolognani wrote:
> > Branch: https://gitlab.com/abologna/libvirt/-/tree/ci-full-gitlab-registry
> > Pipeline: https://gitlab.com/abologna/libvirt/pipelines/150891361
> > 
> > This is what we're already doing with the subprojects we've migrated
> > to GitLab CI and, as of earlier today, all projects under the
> > libosinfo umbrella.
> > 
> > Once this is merged, we can stop publishing container images on Quay
> > and archive the libvirt-dockerfiles repository.
> > 
> > Patch 3/5 has been trimmed in order to comply with the size limits
> > of the mailing list. You can grab the unabridged version with
> > 
> >   $ git fetch https://gitlab.com/abologna/libvirt ci-full-gitlab-registry
> This is a lot of files and lines of text/code. I was wondering about
> building the dockerfiles as part of the container_job_definition.
> To me it seems like a lot of duplication and a lot of noise in the
> future if we decide to change the dockerfiles generation. The main
> difference that I can think of is that with files in repository we
> need to regenerate all the dockerfiles to apply changes made in
> libvirt-ci but with the automatic generation we would have that for
> free.

The key reason for keeping the dockerfiles in the libvirt.git repo and
NOT auto-generating them on the fly is to ensure the CI process is
self-contained, with no dependancy on external moving parts in other
git repos.

If you automatically generated dockerfiles from libvirt-ci.git, then
you end up with unstable CI when changes in libvirt.git need to be
made in lock-step with changes in libvirt-ci.git. If you change
libvirt.git first, CI will break if it runs before libvirt-ci.git
is updated. If you change libvirt-ci.git first, then CI will break
if it runs before libvirt.git is updated. This is a no-win situation.
This is especially painful when you consider that a user's fork of
libvirt.git may not updated to current master. Or consider a user
who needs to make changes to libvirt.git that require updated
dockerfiles and needs to be able to test them before any change
in libvirt-ci.git is present.

We've seen these problems many times with our current Jenkins setup
where CI breaks for a period when we have to do matching updates
between both libvirt-ci.git & libvirt.git.

With dockerfiles kept in libvirt.git we know that the containers
we're building will always contain exactly what we need. This also
makes it easy for users to experiment with changes, as they can
modify the dockerfiles directly to add/remove pieces. Such changes
can be propagated back to libvirt-ci.git out of band.

> Both approaches have some benefits and drawbacks and I guess we could
> have some variable to prevent automatic generation of dockerfiles to
> make sure that unwanted changes in libvirt-ci doesn't affect CI for
> all libvirt repositories, on the other hand it would automatically
> check that changes to libvirt-ci doesn't break anything.

Changes to libvirt-ci that affect the dockerfiles, should come with
a URL pointer to a merge request against the affected project(s),
showing the succesful CI run with updated dockerfiles.

> I personally don't like the need to introduce 5000+ lines just for
> compilation testing.

That's a tiny proportion of the code we have in libvirt.git, so IMHO
it is not worth worrying about. The benefits of having the CI self
contained far outweigh the downside.

Essentially we are prioritizing the main libvirt.git as that is the
primary content from the POV of most contributors.

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

More information about the libvir-list mailing list