GSoC 2022 CI project idea proposal

[Erik Skultety]

On Fri, Feb 25, 2022 at 03:02:49AM -0800, Andrea Bolognani wrote:
> On Fri, Feb 25, 2022 at 11:31:39AM +0100, Erik Skultety wrote:
> > On Thu, Feb 24, 2022 at 06:49:29AM -0800, Andrea Bolognani wrote:
> > > I'm not entirely convinced that requiring the use of lcitool for this
> > > task is necessarily the best idea though. Ideally, it should be
> > > possible for developers to reproduce and debug CI failures locally
> > > without having to clone a second repository. It's fine to require
> > > lcitool for tasks that are performed by maintainers, but casual
> > > contributors should find all they need in libvirt.git itself.
> >
> > Unless they also want to test with VMs as well in which case lcitool is highly
> > recommended. The thing is that right now ci/helper wraps the Makefile. Let's
> > say you (or someone else) replaces it with a python equivalent, I don't think
> > that the number of contributors suddenly starting to use it will differ much
> > compared to the situation right now, but maybe I'm wrong :).
> > My motivation to add the support to lcitool is simple - right now there's an
> > RFC on adopting some infrastructure in order to run TCK-based functional tests
> > as part of GitLab. That's just the first stage, ultimately we want developers
> > to be able to run the same test suite locally without the need to wait for
> > gitlab and since the current design of the above is using nested virt, it is an
> > achievable goal. There was also a suggestion to adopt the Avocado framework for
> > a new test framework which would be largely based on TCK [1].
> > Now, let's get to the point I'm trying to make :) - my expectation/vision is
> > that both the remote and local functional test executions will revolve around
> > lcitool and so eventually with the adoption of *some* framework it is going to
> > become a requirement that libvirt developers do perform functional testing
> > before submitting a patch series. With that said, it makes more sense in my
> > mind to focus on adding the container runtime support to lcitool in order to
> > move towards the bigger goal and have everything consolidated in a single
> > place.
> 
> Alright, that's a pretty solid argument. And even if we ultimately
> decide that we don't want to require using lcitool, once we have a
> project-agnostic Python implementation of this logic it would be
> reasonably straightforward to turn it into a standalone tool similar
> to the current ci/helper, so we have an exit strategy.
> 
> > > Another thing that has been bothering me is that neither 'ci/helper
> > > build' nor 'lcitool build' will actually perform the exact same build
> > > steps as the corresponding CI job, making them only mildly effective
> > > as debugging tools for CI failures. And of course each of these build
> > > recipes is maintained separately, so we have three similar but not
> > > quite identical scripts floating around.
> >
> > Yes, but on its own, I think that is a problem which can be solved separately.
> 
> Agreed that the two tasks are not necessarily depending on each
> other, but there's significant overlap. I think it makes sense to
> rationalize how the build steps for a project are defined and
> maintained as part of adding "build in a local container" support to
> lcitool.
> 
> Do you have any high-level concerns about the ci/build approach I
> vaguely described? The finer details are of course far from being set
> in stone, but I think the overall idea is solid and we should aim for
> it being implemented as we evolve our CI tooling.

No, I think that was a solid proposal, I would probably think of along those
lines as well had I ever come to that idea myself :).
Having each repo define their own build script which can be consumed both
during local test executions and copied to the Dockerfile for a gitlab job to
consume makes complete sense.
Top it off with something like 'lcitool container --script
<path_to_build_script>' and I think we have a solid ground for future work.

Erik