[Patchew-devel] [RFC] Replacing tester with gitlab CI

[Fam Zheng]

On Thu, 2020-10-01 at 13:27 +0200, Paolo Bonzini wrote:
> On 01/10/20 13:02, Fam Zheng wrote:
> > Here is an idea about how we can drop some code and simplify
> > things.
> > 
> > Back when patchew tester was initially designed, there wasn't
> > something
> > as flexible as needed. Today, gitlab-ci has everything we want, so
> > if
> > the applier pushes tags to a gitlab repo, all the testing stuff can
> > happen there.
> 
> Except one thing that we could keep on Patchew testers: gitlab-ci
> doesn't have KVM (though we could use Travis or custom runners) which
> QEMU needs in order to test FreeBSD.  But yeah I totally agree that
> Gitlab or GitHub is a much better alternative to our custom testers.
> 

Yes, KVM is a little tricky, as well as using docker in the tests. That
is exactly the requirements/capabilities mechanism we have. One way to
make it less messy is to put group shared runners under 
https://gitlab.com/qemu-project, and let the applier push to 
https://gitlab.com/qemu-project/ci-hub . This way both the main repo
and the CI repo can share the same pool. The approach applies to any
project that requires its own runners too.

"Bring your own runners" is a nice aspect of both patchew and gitlab,
it's essentially why we can switch. The upside of it is if we are using
gitlab, more people will have a decent understanding of those, and
potentially help build more into CI.

> Time for Patchew 3.0? :)

I don't mind :)

> 
> > Using gitlab-ci, we can change steps 3-6 to:
> > 
> > 3. The applier download mbox and applies as usual.
> > 
> > 4. The applier injects with `include:remote` a {.post: when:always}
> > stage into .gitlab-ci.yml.
> > 
> > 5. The branch, with the modified .gitlab-ci.yml, is tagged and
> > pushed
> > to (for example)
> > 
> >     https://gitlab.com/patchew/ci-hub
> > 
> > 6. Gitlab CI kicks in and runs the tests.
> > 
> > 7. In the end, if any tests defined in the project's original
> > .gitlab-
> > ci.yml failed, the injected .post stage can report to patchew
> > server
> > with existing API, from there we can send notifications etc.
> 
> This could alternatively (and possibly more easily) be a Job webhook
> on
> gitlab.com/patchew/ci-hub.

Yes, there is callbacks for pipeline events:


https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#pipeline-events

The interface looks sane so I believe it can do what we want, but it
doesn't look very easy to embed metadata (e.g. key information we
extract in tests).

And I don't like webhooks in general because the interface is outside
of control, and it's harder to develop or debug with locally. On the
contrary with custom scripts in CI, if anything doesn't work as
expected, it's super easy to figure out what happened in CI logs.

Also I think it's in long term easier if Patchew code can stay 100%
gitlab agnostic. What's missing is, I think, adding a post-apply hook
script for the applier.

Fam

> 
> Paolo
> 
> > Benefits:
> > 
> > * Secrets and credentials can be managed securely with gitlab-ci
> > env
> > var.
> > 
> > * We let projects to define the tests, no more manual testing
> > config at
> > patchew server side.
> > 
> > * Less code for patchew.
> > 
> > * Gitlab provides free runner quota, and even managing gitlab-
> > runners
> > by ourselves is better than managing patchew testers ourselves
> > because
> > it's standard.
> > 
> > * Many things about gitlab CI UI are much better than patchew.org,
> > so
> > it's a great bonus - we can include a link to gitlab.com in the
> > email
> > report. Hopefully that is more familiar to people.
> > 
> > * This model is more scalable and hopefully easier to adopt by
> > other
> > projects, such as Xen and perhaps Linux too.
> > 
> > 
> > Last but not least, we will need to PoC the injected .post stage.
> > 
> > Fam
> > 
> 
> _______________________________________________
> Patchew-devel mailing list
> Patchew-devel at redhat.com
> https://www.redhat.com/mailman/listinfo/patchew-devel
>