[libvirt PATCH v3 3/4] ci: Use GitLab container registry

Daniel P. Berrangé berrange at redhat.com
Wed Jun 10 16:11:49 UTC 2020


On Wed, Jun 10, 2020 at 05:34:13PM +0200, Andrea Bolognani wrote:
> Instead of using pre-built containers hosted on Quay, build
> containers as part of the GitLab CI pipeline and upload them to the
> GitLab container registry for later use.
> 
> This will not significantly slow down builds, because containers are
> only rebuilt when the corresponding Dockerfile has been modified.
> 
> Signed-off-by: Andrea Bolognani <abologna at redhat.com>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ba31afd205..f79f930e66 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -3,6 +3,7 @@ variables:
>  
>  stages:
>    - sanity_checks
> +  - containers
>    - prebuild
>    - native_build
>    - cross_build
> @@ -16,10 +17,43 @@ stages:
>  
>  # Common templates
>  
> +# Containers that are necessary for the default pipeline
> +.container_default_job_template: &container_default_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:$CI_COMMIT_REF_SLUG"
> +    - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt/ci-$NAME:master"

This is different to what we've done on all the other repos. I originally
used this, but noted that it results in a ever growing set of tags being
published in the container registry, as users will have a new branch name
for every piece of work. It also means you'll never a get a cache hit
from the user's registry across feature branches, though that is mitigated
to by fact that we'll consider the global cache too I guess.

> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/ci-$NAME.Dockerfile" ci/containers
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout


> +
> +# Containers that are only needed for the full pipeline
> +.container_extra_job_template: &container_extra_job_definition
> +  <<: *container_default_job_definition
> +  only:
> +    - master
> +    - /^ci-full-.*$/
> +
> +# We build many containers which can be useful to debug problems but are not
> +# needed for the pipeline itself to complete: those sometimes fail, and when
> +# that happens it's mostly because of temporary issues with Debian sid. We
> +# don't want those failures to affect the overall pipeline status
> +.container_optional_job_template: &container_optional_job_definition
> +  <<: *container_extra_job_definition
> +  allow_failure: true
> +
>  # Default native build jobs that are always run
>  .native_build_default_job_template: &native_build_default_job_definition
>    stage: native_build
> -  image: quay.io/libvirt/buildenv-libvirt-$NAME:latest
> +  image: $CI_REGISTRY_IMAGE/ci-$NAME:$CI_COMMIT_REF_SLUG
>    cache:
>      paths:
>        - ccache/
> @@ -67,7 +101,7 @@ stages:
>  # Default cross build jobs that are always run
>  .cross_build_default_job_template: &cross_build_default_job_definition
>    stage: cross_build
> -  image: quay.io/libvirt/buildenv-libvirt-$NAME-cross-$CROSS:latest
> +  image: $CI_REGISTRY_IMAGE/ci-$NAME-cross-$CROSS:$CI_COMMIT_REF_SLUG
>    cache:
>      paths:
>        - ccache/
> @@ -89,6 +123,212 @@ stages:
>      - /^ci-full-.*$/
>  
>  
> +# Native container build jobs
> +
> +x64-centos-7-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: centos-7
> +
> +x64-centos-8-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: centos-8
> +
> +x64-centos-stream-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: centos-stream
> +
> +x64-debian-9-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-9
> +
> +x64-debian-10-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: debian-10
> +
> +x64-debian-sid-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-sid
> +
> +x64-fedora-31-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: fedora-31
> +
> +x64-fedora-32-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: fedora-32
> +
> +x64-fedora-rawhide-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: fedora-rawhide
> +
> +x64-opensuse-151-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: opensuse-151
> +
> +x64-ubuntu-1804-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: ubuntu-1804
> +
> +x64-ubuntu-2004-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: ubuntu-2004
> +
> +
> +# Cross-build containers build jobs
> +
> +aarch64-debian-9-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-9-cross-aarch64
> +
> +armv6l-debian-9-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-9-cross-armv6l
> +
> +armv7l-debian-9-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-9-cross-armv7l
> +
> +mips-debian-9-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-9-cross-mips
> +
> +mips64el-debian-9-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-9-cross-mips64el
> +
> +mipsel-debian-9-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-9-cross-mipsel
> +
> +ppc64le-debian-9-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-9-cross-ppc64le
> +
> +s390x-debian-9-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-9-cross-s390x
> +
> +aarch64-debian-10-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-10-cross-aarch64
> +
> +armv6l-debian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-armv6l
> +
> +armv7-ldebian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-armv7l
> +
> +i686-debian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-i686
> +
> +mips-debian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-mips
> +
> +mips64el-debian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-mips64el
> +
> +mipsel-debian-10-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-10-cross-mipsel
> +
> +ppc64le-debian-10-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-10-cross-ppc64le
> +
> +s390x-debian-10-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: debian-10-cross-s390x
> +
> +aarch64-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-aarch64
> +
> +armv6l-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-armv6l
> +
> +armv7-ldebian-sid-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: debian-sid-cross-armv7l
> +
> +i686-debian-sid-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-sid-cross-i686
> +
> +mips-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-mips
> +
> +mips64el-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-mips64el
> +
> +mipsel-debian-sid-container:
> +  <<: *container_extra_job_definition
> +  variables:
> +    NAME: debian-sid-cross-mipsel
> +
> +ppc64le-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-ppc64le
> +
> +s390x-debian-sid-container:
> +  <<: *container_optional_job_definition
> +  variables:
> +    NAME: debian-sid-cross-s390x
> +
> +mingw32-fedora-rawhide-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: fedora-rawhide-cross-mingw32
> +
> +mingw64-fedora-rawhide-container:
> +  <<: *container_default_job_definition
> +  variables:
> +    NAME: fedora-rawhide-cross-mingw64
> +
> +
>  # Native architecture build + test jobs
>  
>  x64-debian-9:
> @@ -116,6 +356,11 @@ x64-centos-8:
>    variables:
>      NAME: centos-8
>  
> +x64-centos-stream:
> +  <<: *native_build_extra_job_definition
> +  variables:
> +    NAME: centos-stream
> +
>  x64-fedora-31:
>    <<: *native_build_extra_job_definition
>    variables:
> @@ -231,6 +476,7 @@ mingw64-fedora-rawhide:
>  #    https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=website
>  website:
>    stage: prebuild
> +  image: $CI_REGISTRY_IMAGE/ci-centos-8:$CI_COMMIT_REF_SLUG
>    before_script:
>      - *script_variables
>    script:
> @@ -241,7 +487,6 @@ website:
>      - $MAKE -C docs install
>      - cd ..
>      - mv vroot/share/doc/libvirt/html/ website
> -  image: quay.io/libvirt/buildenv-libvirt-centos-8:latest
>    artifacts:
>      expose_as: 'Website'
>      name: 'website'
> @@ -253,6 +498,7 @@ website:
>  
>  codestyle:
>    stage: prebuild
> +  image: $CI_REGISTRY_IMAGE/ci-centos-8:$CI_COMMIT_REF_SLUG
>    before_script:
>      - *script_variables
>    script:
> @@ -260,7 +506,6 @@ codestyle:
>      - cd build
>      - ../autogen.sh || (cat config.log && exit 1)
>      - $MAKE syntax-check
> -  image: quay.io/libvirt/buildenv-libvirt-centos-8:latest
>  
>  
>  # This artifact published by this job is downloaded to push to Weblate
> @@ -268,6 +513,7 @@ codestyle:
>  #    https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potfile
>  potfile:
>    stage: prebuild
> +  image: $CI_REGISTRY_IMAGE/ci-centos-8:$CI_COMMIT_REF_SLUG
>    only:
>      - master
>    before_script:
> @@ -280,7 +526,6 @@ potfile:
>      - $MAKE -C po libvirt.pot
>      - cd ..
>      - cp po/libvirt.pot libvirt.pot
> -  image: quay.io/libvirt/buildenv-libvirt-centos-8:latest
>    artifacts:
>      expose_as: 'Potfile'
>      name: 'potfile'
> diff --git a/ci/containers/README.rst b/ci/containers/README.rst
> new file mode 100644
> index 0000000000..530897e311
> --- /dev/null
> +++ b/ci/containers/README.rst
> @@ -0,0 +1,14 @@
> +CI job assets
> +=============
> +
> +This directory contains assets used in the automated CI jobs, most
> +notably the Dockerfiles used to build container images in which the
> +CI jobs then run.
> +
> +The ``refresh`` script is used to re-create the Dockerfiles using the
> +``lcitool`` command that is provided by repo
> +https://gitlab.com/libvirt/libvirt-ci
> +
> +The containers are built during the CI process and cached in the GitLab
> +container registry of the project doing the build. The cached containers
> +can be deleted at any time and will be correctly rebuilt.
> 
> [... imagine lots and lots of Dockerfiles here ...]
> 
> diff --git a/ci/containers/refresh b/ci/containers/refresh
> new file mode 100755
> index 0000000000..8c00363ae1
> --- /dev/null
> +++ b/ci/containers/refresh
> @@ -0,0 +1,43 @@
> +#!/bin/sh
> +
> +if test -z "$1"
> +then
> +    echo "syntax: $0 PATH-TO-LCITOOL"
> +    exit 1
> +fi
> +
> +LCITOOL=$1
> +
> +if ! test -x "$LCITOOL"
> +then
> +    echo "$LCITOOL is not executable"
> +    exit 1
> +fi
> +
> +HOSTS=$($LCITOOL hosts | grep -v freebsd)
> +
> +for host in $HOSTS
> +do
> +    name=${host#libvirt-}
> +
> +    case "$name" in
> +    fedora-rawhide)
> +        for cross in mingw32 mingw64
> +        do
> +            $LCITOOL dockerfile $host libvirt --cross $cross >ci-$name-cross-$cross.Dockerfile
> +        done
> +        ;;
> +    debian-*)
> +        for cross in aarch64 armv6l armv7l i686 mips mips64el mipsel ppc64le s390x
> +        do
> +            if test "$name" = "debian-9" && test "$cross" = "i686"
> +            then
> +                continue
> +            fi
> +            $LCITOOL dockerfile $host libvirt --cross $cross >ci-$name-cross-$cross.Dockerfile
> +        done
> +        ;;
> +    esac
> +
> +    $LCITOOL dockerfile $host libvirt >ci-$name.Dockerfile

In all the other projects we've just used $host.Dockerfile.

If we really want to eliminate the "libvirt-" prefix, then it is probably
better to modify lcitool to remove this prefix. It is a pretty pointless
name prefix considering the targets are used for non-libvirt projects too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|




More information about the libvir-list mailing list