[libvirt PATCH 00/20] ci: Move GitLab build recipes to a standalone script

[Daniel P. Berrangé]

On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote:
> On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote:
> I appreciate the goal of being able to run CI commands locally, but
> I'm not feeling this approach is a net win. I'd much rather just
> having developers invoke the meson command directly, which is not
> that hard.
> 
> If we do really want to provide something that 100% mirrors the
> gitlab CI job commands though, I'm even more convinced now that
> we should be using the .gitlab-ci.yml as the canonical source.
> 
> Since I last mentioned that idea, I found out that this should
> be something we can achieve via the gitlab runner 'exec' command.

I wonder if we've been thinking about this from the wrong angle.

We've been looking at the problem of "we have a gitlab-ci.yml" and
we want to be able to locally run the jobs it defines. This creates
us the problem of interpreting the gitlab-ci.yml file, which is
very hard.

95% of what we define in the gitlab-ci.yml file comes in via the
includes of ci/gitlab.yml which is entirely auto-generated from
our ci/manifest.yml file. 

IOW, can we rephase the problem to "we have a ci/manifest.yml" and
we want to locally run the jobs it implies. We already have the
code for parsing the ci/manifest.yml file, and so (mostly) know
what jobs we expect to have.

What we're missing is the project specific build rules which
still live in the hand crafted .gitlab-ci.yml.

If we could get the project specific build rules to be expressed
in the ci/manifest.yml file,  then we have all the info we need
to be able to run jobs locally. We could then make even the root
.gitlab-ci.yml be auto-generated, which would be nice.

If we have the project build rules in ci/manifest.yml, we ahve
lots of flexibility. We can easily trigger the builds in containers
or VMs, or the local host, or whatever. 'lcitool' already has the
command for runing builds in VMs, but that relies on build rules
we defined separately in libvirt-ci.git and we've somewhat
neglected the VM build logic since adopting containers.

Essentially ci/manifest.yml would become a fully self-contained
canonical representation of any info needed to run builds, and be
independant of gitlab CI, or any other CI framework we might like
to plug into in the future. So 'lcitool' can be the entrypoint
for triggering builds  that 100% match what gets run in gitlab

Yes, this only addresses the core build/unittest side of things.

The integration test suite is still a separate task to address.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|