GSoC 2022 CI project idea proposal

Fri Feb 25 10:31:39 UTC 2022

On Thu, Feb 24, 2022 at 06:49:29AM -0800, Andrea Bolognani wrote:
> On Tue, Feb 22, 2022 at 05:04:02PM +0100, Erik Skultety wrote:
> > So, we're offloading as much CI configuration/workflow stuff to lcitool
> > as possible. We can generate config files, install/update machines
> > (local or remote), dump Dockerfiles...we can't build and run those containers
> > locally. Instead, we have a CI helper script in libvirt repo which essentially
> > just wraps a Makefile which pulls a gitlab container for you and:
> >     - gives you shell, or
> >     - runs the build and tests
> >
> > I'm not sure how many people actually know we have that helper script let alone
> > use it. I've been playing with the idea that we could integrate what's done in
> > the Makefile to lcitool utilizing either the podman library [1] or the docker
> > library [2]. Apart from consolidating all CI services-related efforts to
> > lcitool the other benefit would be that we could gain the ability to run and
> > debug in a project-specific container also in other libvirt projects not just
> > main libvirt.
> >
> > So, I though this could be a nice project for GSoC. Ideas?
> 
> I've been meaning to replace the make-based logic used for spawning
> local containers with a Python equivalent and get rid of ci/Makefile
> entirely. There was some progress last year, but I got sidetracked
> and never managed to finish the job. So obviously I'm in favor of
> doing more work in that area, especially if it's someone else doing
> it O:-)
> 
> I'm not entirely convinced that requiring the use of lcitool for this
> task is necessarily the best idea though. Ideally, it should be
> possible for developers to reproduce and debug CI failures locally
> without having to clone a second repository. It's fine to require
> lcitool for tasks that are performed by maintainers, but casual
> contributors should find all they need in libvirt.git itself.

Unless they also want to test with VMs as well in which case lcitool is highly
recommended. The thing is that right now ci/helper wraps the Makefile. Let's
say you (or someone else) replaces it with a python equivalent, I don't think
that the number of contributors suddenly starting to use it will differ much
compared to the situation right now, but maybe I'm wrong :).
My motivation to add the support to lcitool is simple - right now there's an
RFC on adopting some infrastructure in order to run TCK-based functional tests
as part of GitLab. That's just the first stage, ultimately we want developers
to be able to run the same test suite locally without the need to wait for
gitlab and since the current design of the above is using nested virt, it is an
achievable goal. There was also a suggestion to adopt the Avocado framework for
a new test framework which would be largely based on TCK [1].
Now, let's get to the point I'm trying to make :) - my expectation/vision is
that both the remote and local functional test executions will revolve around
lcitool and so eventually with the adoption of *some* framework it is going to
become a requirement that libvirt developers do perform functional testing
before submitting a patch series. With that said, it makes more sense in my
mind to focus on adding the container runtime support to lcitool in order to
move towards the bigger goal and have everything consolidated in a single
place.

> 
> Another thing that has been bothering me is that neither 'ci/helper
> build' nor 'lcitool build' will actually perform the exact same build
> steps as the corresponding CI job, making them only mildly effective
> as debugging tools for CI failures. And of course each of these build
> recipes is maintained separately, so we have three similar but not
> quite identical scripts floating around.

Yes, but on its own, I think that is a problem which can be solved separately.

> 
> Here's my (fairly hand-wavy :) idea of how things should work.
> 
> Each project's repository should contain its build steps in the form
> of a ci/build script. The exact calling interface for this script
> will have to be determined, but based on existing usage at the very
> least we need to be able to specify the build type (regular, cross
> and with dependencies built from git) and target-specific settings
> (for example whether to skip building RPMs). So I can imagine the
> invocation looking something like
> 
>   $ TYPE=native RPMS=skip ci/build
> 
> The build steps that currently live in the .native_build_job,
> .cross_build_job and .native_git_build_job templates in
> .gitlab-ci.yml will all be moved to the ci/build script. It shouldn't
> be necessary to alter them significantly in the process.
> 
> With this interface defined, we can change 'lcitool manifest' so that
> the jobs it generates go from
> 
>   x86_64-almalinux-8-clang:
>     extends: .native_build_job
>     needs:
>       - x86_64-almalinux-8-container
>     variables:
>       CC: clang
>       NAME: almalinux-8
>       RPM: skip
> 
> to
> 
>   x86_64-almalinux-8-clang:
>     needs:
>       - x86_64-almalinux-8-container
>     script:
>       - TYPE=native NAME=almalinux-8 CC=clang RPM=skip ci/build
> 
> Of course it would still be possible to tell lcitool to use a custom
> template for the job instead, which might be necessary for setting up
> CI caching and the like. For simple cases though, you'd be able to
> use the default implementation.
> 
> 'lcitool build' would also be changed so that it invokes the ci/build
> script in the project's repository.
> 
> The last missing piece would then be to finish converting the
> ci/helper script that exists in libvirt to Python and make it so
> calling that entry point also ultimately in running the ci/build
> script.
> 
> Of course having multiple copies of the ci/helper script and its
> logic for listing / running containers around is problematic because
> they would get out of sync over time... Perhaps we can maintain that
> script as part of libvirt-ci.git, and generate a local copy to be
> stored in each project's repository at the same time as other
> CI-related files such as Dockerfiles are generated? That sounds like
> it could work, but admittedly this is the fuzziest part of the entire
> plan :)

The thing is that the GSoC project evaluation process has already begun, so I'd
like to have something listed, unless you completely disagree with the vision I
tried to lay out above in which case I'll have to retract on the idea
completely and it'll require further discussion on the direction we want to
take (even for the testing framework's sake in the other thread).

[1] https://listman.redhat.com/archives/libvir-list/2021-July/msg00028.html

Erik