[Pulp-dev] Moving to Github Actions

Wed Feb 12 16:58:57 UTC 2020

bringing in some data about CI,

Last month, we had a considerable increase in total builds and in queue
time:
[image: image.png]

                                   https://travis-ci.org/pulp?tab=insights

[image: image.png]

                                https://travis-ci.com/pulp?tab=insights

Best regards,
Fabricio Aguiar
Software Engineer, Pulp Project
Red Hat Brazil - Latam <https://www.redhat.com/>
+55 11 999652368

On Tue, Feb 11, 2020 at 3:35 PM Mike DePaulo <mikedep333 at redhat.com> wrote:

> Here are my set of thoughts on many things mentioned.
>
> TL;DR: We still need to run CI on CentOS/Fedora, but using cloud instances
> of CentOS/Fedora (interacted with via SSH/Ansible from the GHA Ubuntu
> client VM) might be preferable to using Fedora CI for certain tests.
>
> 1. "We should test GHA via the ansible-pulp related repos now, and then
> come up with a thorough & quick schedule to migrate from Travis to GHA
> entirely, resources permitting."
> I totally agree with this.
>
> 2. "We must use Fedora/Centos CI for SELinux policy testing at all,
> because Travis & GHA use Ubuntu, whose kernel doesn't support SELinux."
> I do not think this is correct. I've researched this, but haven't test it.
> SELinux upstream seems to run their CI on Ubuntu:
> https://github.com/SELinuxProject/selinux/blob/master/.travis.yml
>
> https://github.com/SELinuxProject/selinux-testsuite/blob/master/.travis.yml#L53
> How did they make this work?
> My 1 theory is that Ubuntu's kernel has support for both SELinux and
> AppArmor, but Travis slims down the image so that AppArmor does not get
> enabled like on a default Ubuntu install. So SELinux can be enabled at
> runtime.
> My 2nd theory is that enough of a shim (the 2nd link on fake-selinuxfs in
> particular) is sufficient to avoid reboot.
> However, #4 still negates this option.
>
> 3. "We must use Fedora/CentOS CI because pulp-certguard's dependencies are
> an issue on Ubuntu"
> The plugin-template uses our Fedora containers w/ pulp-operator & k3s.
> Isn't this sufficient?
>
> 4.  "We must use Fedora/Centos CI for SELinux policy testing with pulp-rpm
> / pulp-certguard. Containers will provide the pulp-rpm / pulp-container
> deps on ubuntu, but break SELinux testing because SELinux wraps around the
> entire container."
> This is a bigger concern. My research confirms this, even lxc does not
> support SELinux policies *within* the container. Our SELinux policies
> currently support the plugins Katello is integrating: pulp-container,
> pulp-file, pulp-rpm, and pulp-certguard. We could still do CI testing of
> the 1st 2 on Ubuntu though.
>
> 5. "Our CI must run be either capable of running entirely on CentOS/Fedora
> CI, or have only certain tests run on them."
> I am more in favor of the latter, but there is another possible solution
> for SELinux testing. We could have an Amazon EC2 account, openstack
> account, etc that GHA or Travis calls out to. Ansilbe molecule has drivers
> for many cloud compute types:
> https://molecule.readthedocs.io/en/2.22/configuration.html#driver
> It would create a Fedora / CentOS instance specifically for testing via
> cloud APIs, and then run our ansible installer (SSH) against it from the
> GHA / Travis instance, then delete it at the end. It would mean that the
> Pulp Project would still have no persistent infrastructure.
>
> 6. On using Fedora with their Zuul CI instance:
> This looks promising, but having read their PDF, I am concerned that
> Fedora's instance would be specifically configured for their use case in
> ways that we can reconfigure. Hopefully we can. Some of their use case is
> integration with pagure rather than GitHub, Koji artifact storage
> integration, etc:
>
> https://fedoraproject.org/w/uploads/1/1e/CI_CD_for_Fedora_packaging_with_Zuul_-_final_-_with_notes.pdf
>
> -Mike
>
> On Mon, Feb 10, 2020 at 8:29 AM David Davis <daviddavis at redhat.com> wrote:
>
>> Is this the Software Factory instance of Zuul[0]? I can reach out to them
>> and see if it would make sense as an option for Pulp.
>>
>> [0] https://softwarefactory-project.io/zuul/t/local/status
>>
>> David
>>
>>
>> On Sun, Feb 9, 2020 at 11:51 AM Neal Gompa <ngompa13 at gmail.com> wrote:
>>
>>> On Sun, Feb 9, 2020 at 9:46 AM David Davis <daviddavis at redhat.com>
>>> wrote:
>>>
>>>> Thanks Brian and Daniel. I agree on the points you both raised.
>>>>
>>>> Brian, to you specific questions/points:
>>>>
>>>> ## We need details on each piece of the Travis workflow, where it will
>>>> be ported to, and a rough estimate of how long each piece would take. I
>>>> think these things would make a great EPIC.
>>>>
>>>> I have a Github Actions epic. I plan to update it this week based on
>>>> our conversation and will add more specific details, estimates, etc. I'll
>>>> respond when it's ready for review.
>>>>
>>>> https://pulp.plan.io/issues/6065
>>>>
>>>>
>>>> ## Who will work on it? It needs I think 2 fully dedicated people who
>>>> already completely understand the Travis stuff in detail. It's too hard for
>>>> one person and would take too long...
>>>>
>>>> I definitely agree we need at least 2 people to work on this. We need
>>>> as many people as possible to understand Github Actions.
>>>>
>>>> I don't know who has time for this right now. I imagine it'll probably
>>>> have to wait until next sprint (Sprint 67). Or at least I personally won't
>>>> have time until next week at the earliest. That'll give us time to plan
>>>> though.
>>>>
>>>> In the meantime, I'd consider letting the installer team merge
>>>> Fabricio's ansible-pulp PR[0]. This will also alleviate much of the
>>>> immediate need and let us begin collecting real world data/experience as
>>>> well.
>>>>
>>>>
>>> Has anyone reached out to the Fedora CI team about using their Zuul
>>> instance? Perhaps they've got an easy automated process for using that on
>>> projects. Zuul can spin up either Fedora or CentOS environments, which
>>> should satisfy the need for being able to test esoteric things like FIPS
>>> mode while also being able to get fresh environments and dependencies
>>> through Fedora environments.
>>>
>>> It may be better to consider using Fedora CI over CentOS CI due to the
>>> better system overall, too...
>>>
>>>
>>>
>>> --
>>> 真実はいつも一つ！/ Always, there's only one truth!
>>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat <https://www.redhat.com/>
>
> IM: mikedep333
>
> GPG: 51745404
> <https://www.redhat.com/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200212/01ed85ac/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 62204 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200212/01ed85ac/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 56688 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200212/01ed85ac/attachment-0001.png>