[libvirt PATCH 4/5] gitlab: add several native CI jobs

[Erik Skultety]

On Mon, Mar 23, 2020 at 12:01:49PM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 23, 2020 at 12:50:45PM +0100, Erik Skultety wrote:
> > On Fri, Mar 20, 2020 at 03:31:24PM +0000, Daniel P. Berrangé wrote:
> > > On Fri, Mar 20, 2020 at 04:18:58PM +0100, Erik Skultety wrote:
> > > > On Fri, Mar 20, 2020 at 02:59:36PM +0000, Daniel P. Berrangé wrote:
> > > > > On Fri, Mar 20, 2020 at 03:52:15PM +0100, Erik Skultety wrote:
> > > > > > On Tue, Mar 10, 2020 at 10:09:44AM +0000, Daniel P. Berrangé wrote:
> > > > > > > With GitLab CI aiming to replace Jenkins and Travis for CI purposes, we
> > > > > > > need to expand the coverage to include native builds. This patch adds
> > > > > > > all the jobs currently run in Travis. Compared to Jenkins we obviously
> > > > > > > miss the FreeBSD jobs, but also Debian 10 and Fedora 30, but we gain the
> > > > > > > Ubuntu 1804 job as a substitute for Debian.
> > > > > > >
> > > > > > > Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> > > > > > > ---
> > > > > > >  .gitlab-ci.yml | 41 +++++++++++++++++++++++++++++++++++++++++
> > > > > > >  1 file changed, 41 insertions(+)
> > > > > > >
> > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > index e28ec584ea..3e15d08d17 100644
> > > > > > > --- a/.gitlab-ci.yml
> > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > @@ -1,7 +1,39 @@
> > > > > > >  stages:
> > > > > > >    - website
> > > > > > > +  - native_build
> > > > > > >    - cross_build
> > > > > > >
> > > > > > > +
> > > > > > > +.native_build_job_template: &native_build_job_definition
> > > > > > > +  stage: native_build
> > > > > > > +  script:
> > > > > > > +    - mkdir build
> > > > > > > +    - cd build
> > > > > > > +    - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1)
> > > > > > > +    - make -j $(getconf _NPROCESSORS_ONLN) syntax-check
> > > > > > > +    - make -j $(getconf _NPROCESSORS_ONLN) distcheck
> > > > > >
> > > > > > I think ^this should more closely follow what we have in the lcitool playbooks,
> > > > > > e.g. start with:
> > > > > >     - rm -rf build
> > > > >
> > > > > The source tree is already pristine because this is always executed in
> > > > > a fresh container environment, so there's nothing that will need deleting.
> > > >
> > > > Right, but my point was that if e.g. we introduce a FreeBSD builder, we'd want
> > > > to reference the same job template in which case the directory will exist.
> > >
> > > I've not looked in great detail about how gitlab runners work, but something
> > > on the runner needs to be responsible for checking out the git repo before
> > > any of this gitlab-ci.yml scrpit runs.
> > >
> > > IMHO that can ensure it is "git clean -xdf" so that its state matches what
> > > we see with the built-in shared runners build environment.
> >
> > This is achieved by the GIT_CLEAN_FLAGS which, if unspecified, defaults to
> > -ffdx, so we're covered there.
> >
> > One more thing that crossed my mind and may speedup the builds a bit - some
> > time ago I proposed a patch against libvirt-jenkins-ci to enable shallow
> > cloning which was rejected because of compelling reasons. However, for gitlab
> > purposes, we can't really do any debugging on the shared runners, so I believe
> > we may want to apply shallow cloning there, especially since libvirt is quite a
> > large repo. Gitlab claims [1] the default strategy to be 'fetch' if the
> > worktree can be found on the disk (this is when used with the 'docker' executor
> > which should be the case with shared runners). What I couldn't find is how long
> > are these worktrees cached which could therefore be a factor and we might want
> > to favour the shallow clones instead.
> >
> > [1] https://docs.gitlab.com/ee/ci/large_repositories/#git-strategy
>
> I wonder if GitLab has the same problem with shallow cloning as we hit
> before. I can't remember if it was with Travis or Jenkins now, but there
> was a race where it was schedule the job, but before the git repo was
> checked out someone would push another series.  The shallow clone would
> thus get the commit of the newly pushed series, and would not have enough
> history to get to the commit that actually needed testing.
>
> I guess we could get a win even if we set the clone depth to something
> fairly large like 100 commits of history.

Google didn't return anything relevant in this regard, except of a few
resources:

[1] https://docs.gitlab.com/ce/ci/pipelines/settings.html#git-shallow-clone
    Although ^this doesn't apply to libvirt
[2] https://gitlab.com/gitlab-org/gitlab-runner/issues/3460
    ^This I believe would be the long term solution, but I wouldn't hold my
    breath

I'm okay with anything shorter than the full history, branches and tags.