[libvirt PATCH 4/4] gitlab-ci: Introduce a new test 'integration' pipeline stage
Daniel P. Berrangé
berrange at redhat.com
Thu Feb 10 09:40:45 UTC 2022
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
> Create an integration child pipeline in this stage which will trigger a
> multi-project CI build of Perl bindings which are required by the TCK
> test suite.
> In general, this stage will install all the necessary build artifacts
> and configure logging on the worker node prior to executing the actual
> test suite. In case of a failure, libvirt and Avocado logs are saved
> and published as artifacts.
>
> Signed-off-by: Erik Skultety <eskultet at redhat.com>
> ---
> .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++
Sine we have a free choice of naming in this case,
how about we put it in ci/integration.yml
> diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml
> new file mode 100644
> index 0000000000..cabefc5166
> --- /dev/null
> +++ b/.gitlab-ci-integration.yml
> @@ -0,0 +1,116 @@
> +stages:
> + - bindings
> + - integration
> +
> +.tests:
> + stage: integration
> + before_script:
> + - mkdir "$SCRATCH_DIR"
> + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
> + - sudo pip3 install --prefix=/usr avocado-framework
I'd prefer it if we can just 'dnf install avocado' on Fedora at least,
so that we validate that if someone is running avocado locally we're
compatible with what's packaged.
> + - source /etc/os-release # in order to query the vendor-provided variables
> + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 ||
> + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35;
> + then
> + DAEMONS="libvirtd virtlogd virtlockd";
> + else
> + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd";
> + fi
> + - for daemon in $DAEMONS;
> + do
> + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf;
> + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf;
> + sudo systemctl --quiet stop ${daemon}.service;
> + sudo systemctl restart ${daemon}.socket;
> + done
> + - sudo virsh net-start default &>/dev/null || true;
> +
> + script:
> + - mkdir logs
> + - cd "$SCRATCH_DIR"
> + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git
> + - cd libvirt-tck
> + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado
> + after_script:
> + - if test -e "$SCRATCH_DIR"/avocado;
> + then
> + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado;
> + fi
> + - sudo mv /var/log/libvirt logs/libvirt
> + - sudo chown -R $(whoami):$(whoami) logs
> + variables:
> + SCRATCH_DIR: "/tmp/scratch"
> + artifacts:
> + name: logs
> + paths:
> + - logs
> + when: on_failure
> +
> +
> +libvirt-perl-bindings:
> + stage: bindings
> + trigger:
> + project: eskultety/libvirt-perl
> + branch: multi-project-ci
> + strategy: depend
IIUC, what this does is to spawn a pipeline in the
'libvirt-perl' project on the 'multi-project-ci'
branch. Normally this is asyncronous, but
because of the 'strategy: depend' that causes us
to block until this async pipeline is complete.
> +centos-stream-8-tests:
> + extends: .tests
> + needs:
> + - libvirt-perl-bindings
So this triggers the perl bindings pipeline build
> + - pipeline: $PARENT_PIPELINE_ID
> + job: x86_64-centos-stream-8
This is making us wait for the centos-tream-8
job in the normal CI pipeline.
Should we need 'artifacts: true' here too ?
IIRC, artifacts true was the default, but it
feels sane to make it explicit, especially
since you were explicit for the libvirt-perl
job below
> + - project: eskultety/libvirt-perl
> + job: x86_64-centos-stream-8
> + ref: multi-project-ci
> + artifacts: true
And this making us wait for the cento-stream-8
job in te lbivirt-perl pipeline that we spawned
in the 'libvirt-perl-bindings'. IIUC there should
be no waiting needed since 'libvirt-perl-bindings'
was blocking on completion of the trigger pipeline,
so this effectivectly justpulls in the artifacts from
the already finishd job.
I presume your eskultety/libvirt-perl repo
multi-project-ci branch has a cyhange that causes
the perl-Sys-Virt RPMs to be publish as artifacts,
similar to your change in the previous patch in
this series.
> + variables:
> + DISTRO: centos-stream-8
> + tags:
> + - centos-stream-vm
This means none of these jobs will run by default, unless
you're registered a custom runner with this tag. So no
forks will get these jobs, it'll be post-merge only.
That's fine for now. When we switch to merge requests
we will gain ability to trigger this even for forks without
runners as merge request jobs will attach to the primary
project runners, which is nice.
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 4bcaf22ce2..453472c8be 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -4,6 +4,7 @@ variables:
> stages:
> - containers
> - builds
> + - test
> - sanity_checks
Perhaps just put this in the 'sanity_checks' stage ? Since all
the real jobs are in the child pipeline, I don't think we need
an extra stage here.
> .script_variables: &script_variables |
> @@ -128,3 +129,16 @@ coverity:
> - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID"
> rules:
> - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN"
> +
> +integration:
> + stage: test
> + needs:
> + - x86_64-centos-stream-8
> + - x86_64-centos-stream-9
> + - x86_64-fedora-34
> + - x86_64-fedora-35
Ok, so any job in the child pipeline with a dependancy on a
job in this pipeline needs to have the dependancy repeated
here.
> + trigger:
> + include: .gitlab-ci-integration.yml
> + strategy: depend
> + variables:
> + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
More information about the libvir-list
mailing list