[libvirt PATCH v2 1/1] cirrus: Generate jobs dynamically

Erik Skultety eskultet at redhat.com
Tue Jun 30 14:01:50 UTC 2020


On Mon, Jun 29, 2020 at 08:51:51PM +0200, Andrea Bolognani wrote:
> Instead of having static job definitions for FreeBSD and macOS,
> use a generic template for both and fill in the details that are
> actually different, such as the list of packages to install, in
> the GitLab CI job, right before calling cirrus-run.
>
> The target-specific information are provided by lcitool, so that
> keeping them up to date is just a matter of running the refresh
> script when necessary.
>
> Signed-off-by: Andrea Bolognani <abologna at redhat.com>
> ---
>  .gitlab-ci.yml                    | 36 ++++++++++++++-
>  ci/cirrus/build.yml               | 26 +++++++++++
>  ci/cirrus/freebsd-12.yml.j2       | 73 -------------------------------
>  ci/cirrus/libvirt-freebsd-12.vars |  7 +++
>  ci/cirrus/libvirt-macos-1015.vars |  7 +++
>  ci/cirrus/macos-1015.yml.j2       | 38 ----------------
>  ci/cirrus/refresh                 | 22 ++++++++++
>  7 files changed, 97 insertions(+), 112 deletions(-)
>  create mode 100644 ci/cirrus/build.yml
>  delete mode 100644 ci/cirrus/freebsd-12.yml.j2
>  create mode 100644 ci/cirrus/libvirt-freebsd-12.vars
>  create mode 100644 ci/cirrus/libvirt-macos-1015.vars
>  delete mode 100644 ci/cirrus/macos-1015.yml.j2
>  create mode 100755 ci/cirrus/refresh
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e6eb2f9905..5565750b7e 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -58,11 +58,35 @@ stages:
>  # Jobs that we delegate to Cirrus CI because they require an operating
>  # system other than Linux. These jobs will only run if the required
>  # setup has been performed on the GitLab account (see ci/README.rst).
> +#
> +# The Cirrus CI configuration is generated by replacing target-specific
> +# variables in a generic template: some of these variables are provided
> +# when the GitLab CI job is defined, others are taken from a shell
> +# snippet generated using lcitool.
> +#
> +# Note that the $PATH environment variable has to be treated with
> +# special care, because we can't just override it at the GitLab CI job
> +# definition level or we risk breaking it completely.

Informative, thanks :). I just wish I didn't have to google what semantics the
various types of parameter expansion in Bash had (obviously not your fault).

>  .cirrus_build_job_template: &cirrus_build_job_definition
>    stage: builds
>    image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>    script:
> -    - cirrus-run ci/cirrus/$NAME.yml.j2
> +    - source ci/cirrus/libvirt-$NAME.vars
> +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> +          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
> +          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
> +          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
> +          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
> +          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
> +          -e "s|[@]PKGS@|$PKGS|g"
> +          -e "s|[@]MAKE@|$MAKE|g"
> +          -e "s|[@]PYTHON@|$PYTHON|g"
> +      <ci/cirrus/build.yml >ci/cirrus/$NAME.yml
> +    - cat ci/cirrus/$NAME.yml

was ^this 'cat' just for debugging purposes or was that intended to be part of
the job output in production?

...

> diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh
> new file mode 100755
> index 0000000000..b84910a645
> --- /dev/null
> +++ b/ci/cirrus/refresh
> @@ -0,0 +1,22 @@
> +#!/bin/sh
> +
> +if test -z "$1"
> +then
> +    echo "syntax: $0 PATH-TO-LCITOOL"
> +    exit 1
> +fi
> +
> +LCITOOL=$1
> +
> +if ! test -x "$LCITOOL"
> +then
> +    echo "$LCITOOL is not executable"
> +    exit 1
> +fi
> +
> +HOSTS=$($LCITOOL hosts | grep -E 'freebsd-12|macos')
> +
> +for host in $HOSTS
> +do
> +    $LCITOOL dockerfile "$host" libvirt --variables >"$host.vars"
> +done

Overall, I prefer this to v1 because of the further level of abstraction this
introduces. However, the final format we're producing here is a YAML template
rather than a Dockerfile, so introducing just an option for the "dockerfile"
subcommand rather than a dedicated "cirrus-ci-file" (or similar) subcommand
doesn't feel completely right, especially from the long run perspective. At
some point we'd like to generate the whole gitlab-ci.yml file, so that one
would very likely get a dedicated subcommand as well - it should, as it's just
another "plugin" generator. If we expand beyond libvirt, say QEMU, we may need
to add a different CI provider plugin as well.

Regards,
Erik




More information about the libvir-list mailing list