GSoC 2022 CI project idea proposal

[Andrea Bolognani]

On Fri, Feb 25, 2022 at 11:31:39AM +0100, Erik Skultety wrote:
> On Thu, Feb 24, 2022 at 06:49:29AM -0800, Andrea Bolognani wrote:
> > I'm not entirely convinced that requiring the use of lcitool for this
> > task is necessarily the best idea though. Ideally, it should be
> > possible for developers to reproduce and debug CI failures locally
> > without having to clone a second repository. It's fine to require
> > lcitool for tasks that are performed by maintainers, but casual
> > contributors should find all they need in libvirt.git itself.
>
> Unless they also want to test with VMs as well in which case lcitool is highly
> recommended. The thing is that right now ci/helper wraps the Makefile. Let's
> say you (or someone else) replaces it with a python equivalent, I don't think
> that the number of contributors suddenly starting to use it will differ much
> compared to the situation right now, but maybe I'm wrong :).
> My motivation to add the support to lcitool is simple - right now there's an
> RFC on adopting some infrastructure in order to run TCK-based functional tests
> as part of GitLab. That's just the first stage, ultimately we want developers
> to be able to run the same test suite locally without the need to wait for
> gitlab and since the current design of the above is using nested virt, it is an
> achievable goal. There was also a suggestion to adopt the Avocado framework for
> a new test framework which would be largely based on TCK [1].
> Now, let's get to the point I'm trying to make :) - my expectation/vision is
> that both the remote and local functional test executions will revolve around
> lcitool and so eventually with the adoption of *some* framework it is going to
> become a requirement that libvirt developers do perform functional testing
> before submitting a patch series. With that said, it makes more sense in my
> mind to focus on adding the container runtime support to lcitool in order to
> move towards the bigger goal and have everything consolidated in a single
> place.

Alright, that's a pretty solid argument. And even if we ultimately
decide that we don't want to require using lcitool, once we have a
project-agnostic Python implementation of this logic it would be
reasonably straightforward to turn it into a standalone tool similar
to the current ci/helper, so we have an exit strategy.

> > Another thing that has been bothering me is that neither 'ci/helper
> > build' nor 'lcitool build' will actually perform the exact same build
> > steps as the corresponding CI job, making them only mildly effective
> > as debugging tools for CI failures. And of course each of these build
> > recipes is maintained separately, so we have three similar but not
> > quite identical scripts floating around.
>
> Yes, but on its own, I think that is a problem which can be solved separately.

Agreed that the two tasks are not necessarily depending on each
other, but there's significant overlap. I think it makes sense to
rationalize how the build steps for a project are defined and
maintained as part of adding "build in a local container" support to
lcitool.

Do you have any high-level concerns about the ci/build approach I
vaguely described? The finer details are of course far from being set
in stone, but I think the overall idea is solid and we should aim for
it being implemented as we evolve our CI tooling.

-- 
Andrea Bolognani / Red Hat / Virtualization