[libvirt PATCH 4/5] gitlab: add several native CI jobs

[Daniel P. Berrangé]

On Mon, Mar 23, 2020 at 12:50:45PM +0100, Erik Skultety wrote:
> On Fri, Mar 20, 2020 at 03:31:24PM +0000, Daniel P. Berrangé wrote:
> > On Fri, Mar 20, 2020 at 04:18:58PM +0100, Erik Skultety wrote:
> > > On Fri, Mar 20, 2020 at 02:59:36PM +0000, Daniel P. Berrangé wrote:
> > > > On Fri, Mar 20, 2020 at 03:52:15PM +0100, Erik Skultety wrote:
> > > > > On Tue, Mar 10, 2020 at 10:09:44AM +0000, Daniel P. Berrangé wrote:
> > > > > > With GitLab CI aiming to replace Jenkins and Travis for CI purposes, we
> > > > > > need to expand the coverage to include native builds. This patch adds
> > > > > > all the jobs currently run in Travis. Compared to Jenkins we obviously
> > > > > > miss the FreeBSD jobs, but also Debian 10 and Fedora 30, but we gain the
> > > > > > Ubuntu 1804 job as a substitute for Debian.
> > > > > >
> > > > > > Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> > > > > > ---
> > > > > >  .gitlab-ci.yml | 41 +++++++++++++++++++++++++++++++++++++++++
> > > > > >  1 file changed, 41 insertions(+)
> > > > > >
> > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > index e28ec584ea..3e15d08d17 100644
> > > > > > --- a/.gitlab-ci.yml
> > > > > > +++ b/.gitlab-ci.yml
> > > > > > @@ -1,7 +1,39 @@
> > > > > >  stages:
> > > > > >    - website
> > > > > > +  - native_build
> > > > > >    - cross_build
> > > > > >
> > > > > > +
> > > > > > +.native_build_job_template: &native_build_job_definition
> > > > > > +  stage: native_build
> > > > > > +  script:
> > > > > > +    - mkdir build
> > > > > > +    - cd build
> > > > > > +    - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1)
> > > > > > +    - make -j $(getconf _NPROCESSORS_ONLN) syntax-check
> > > > > > +    - make -j $(getconf _NPROCESSORS_ONLN) distcheck
> > > > >
> > > > > I think ^this should more closely follow what we have in the lcitool playbooks,
> > > > > e.g. start with:
> > > > >     - rm -rf build
> > > >
> > > > The source tree is already pristine because this is always executed in
> > > > a fresh container environment, so there's nothing that will need deleting.
> > >
> > > Right, but my point was that if e.g. we introduce a FreeBSD builder, we'd want
> > > to reference the same job template in which case the directory will exist.
> >
> > I've not looked in great detail about how gitlab runners work, but something
> > on the runner needs to be responsible for checking out the git repo before
> > any of this gitlab-ci.yml scrpit runs.
> >
> > IMHO that can ensure it is "git clean -xdf" so that its state matches what
> > we see with the built-in shared runners build environment.
> 
> This is achieved by the GIT_CLEAN_FLAGS which, if unspecified, defaults to
> -ffdx, so we're covered there.
> 
> One more thing that crossed my mind and may speedup the builds a bit - some
> time ago I proposed a patch against libvirt-jenkins-ci to enable shallow
> cloning which was rejected because of compelling reasons. However, for gitlab
> purposes, we can't really do any debugging on the shared runners, so I believe
> we may want to apply shallow cloning there, especially since libvirt is quite a
> large repo. Gitlab claims [1] the default strategy to be 'fetch' if the
> worktree can be found on the disk (this is when used with the 'docker' executor
> which should be the case with shared runners). What I couldn't find is how long
> are these worktrees cached which could therefore be a factor and we might want
> to favour the shallow clones instead.
> 
> [1] https://docs.gitlab.com/ee/ci/large_repositories/#git-strategy

I wonder if GitLab has the same problem with shallow cloning as we hit
before. I can't remember if it was with Travis or Jenkins now, but there
was a race where it was schedule the job, but before the git repo was
checked out someone would push another series.  The shallow clone would
thus get the commit of the newly pushed series, and would not have enough
history to get to the commit that actually needed testing.

I guess we could get a win even if we set the clone depth to something
fairly large like 100 commits of history.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|