[libvirt PATCH 4/5] gitlab: add several native CI jobs

[Andrea Bolognani]

On Fri, 2020-03-20 at 17:44 +0000, Daniel P. Berrangé wrote:
> On Fri, Mar 20, 2020 at 06:13:13PM +0100, Andrea Bolognani wrote:
> > On Tue, 2020-03-10 at 10:09 +0000, Daniel P. Berrangé wrote:
> > > With GitLab CI aiming to replace Jenkins and Travis for CI purposes, we
> > > need to expand the coverage to include native builds. This patch adds
> > > all the jobs currently run in Travis. Compared to Jenkins we obviously
> > > miss the FreeBSD jobs, but also Debian 10 and Fedora 30, but we gain the
> > > Ubuntu 1804 job as a substitute for Debian.
> > 
> > Once again, two questions:
> > 
> >   * why are we replicating the jobs that already exist in Travis
> >     CI? I get it that we eventually want to move everything over to
> >     GitLab CI, but for the time being it looks like we're not really
> >     gaining anything from culling jobs - we're just significantly
> >     reducing our test coverage;
> > 
> >   * what's the endgame here? Are we going to rely solely on the
> >     runners that GitLab provides for free, or are we going to plug
> >     our own runners into the infrastructure? Because the former
> >     would severely limit our test coverage, and bring it down to a
> >     much, much worse state than what we currently have.
> 
> The goal is to consolidate everything onto GitLab CI so that we have
> a single source of truth for people to look at to determine build
> status. IOW, the end game is to eliminate both Jenkins and Travis.
> 
> The only technical gaps that GitLab has compared to our current
> platforms are FreeBSD and macOS. Linux coverage is fine since it
> can run arbitrary containers.

But we are limited in the number of containers we can run for
each build.

> For FreeBSD we have virtual hardware we can use to provide custom
> runners for GitLab to enable use of VMs with FreeBSD installs.

Since we're going to need a FreeBSD machine configured as a GitLab
runner, wouldn't it make sense for us to also create one or more
Linux machines that would run jobs on behalf of GitLab CI? That way
we'd remove the limitation on the number of containers we can have
running at the same time, and still have everything controlled by
GitLab.

> I'm not proposing to turn off the Linux Jenkins jobs just yet,
> as the sub-package chain builds still rely on that. The sub-packages
> will all need wiring up into GitLab CI too. Ultimately though there's
> nothing Jenkins does that we can't replicate in GitLab.
> 
> We don't have an immediate solution for macOS due to the licensing
> issues around use of macOS, so Travis will unfortunately still be
> needed for that platform, but the remainder of jobs are not required.
> It might be possible to wire up a GitLab job that pushes the changes
> over to Travis, and pulls its build result back for macOS, so we still
> have a single viewing point, even if we still use Travis.

That sounds like it would get very hairy, very quickly. But it would
surely be nice, and an improvement over our current status quo, to
have a consolidated view of all build jobs.

> This series has cut down on the number of cross-build jobs we're running,
> but my experience of looking at the results we've seen with them, is that
> they've almost never identified a bug we didn't already see with one
> for the other jobs. So I think it is reasonable to cut cross-build jobs
> down to focus on the important unique aspects - a 32-bit build and a
> big endian build.
> 
> Overall we're not loosing any notable CI coverage with this series, and
> we'll gain an improved view of our results.

While, for example, Ubuntu 18.04 is an acceptable enough stand-in
for Debian 10, I'd really much rather test on the real thing.

I'm okay with this kind of cutback in the short term, but only if
the long term plan is to restore (and hopefully increase!) coverage,
and I don't see how that can be achieved without bringing up our own
Linux runners.

-- 
Andrea Bolognani / Red Hat / Virtualization