[Pulp-dev] Using nested virtualization for SELinux/FIPS CI testing on Travis & GHA

[Mike DePaulo]

Nested hardware virtualization with KVM works on Travis, but not on GitHub
actions.

I highly recommend we use it, even if it means maintaining some CI tests on
travis rather than GHA.

I evaluated 2 other possible solutions on GitHub Actions: VirtualBox
software virtualization, and qemu emulation.

I tested by creating prototype CI for Pulplift, and after spending many
hours, had to resort to some manual testing of the other 2 alternatives'
final steps. They do work though; they launch virtual/emulated machines.
https://github.com/pulp/pulplift/pull/66

Test results:

Nested hardware virtualization with KVM works on Travis:
Pros:
- Fastest
- Very well featured
- Works with pulplift as-is
Cons:
- Cannot work on GHA, which use Azure legacy BIOS VMs, without nested Intel
VT-X / AMD-V, as of Ubuntu 18.04. (Hoping for change with 20.04).
- May not work on Travis indefinitely. They use LXD now, but have no
guarantees to keep using LXD with Intel VT-x / AMD-V enabled.

Software Virtualization with VirtualBox:
Pros:
- None really, just less slow than qemu
Cons:
- Limited to 32-bit x86 guest OS's. CentOS 7 32-bit x86 is unofficial, and
Fedora 30 is the last 32-bit x86 Fedora. Neither are ideal for testing
- Slow: 1 vCPU, and slower than it was in the past (~2/3 of native CPU
performance) back in 2006 or so when OS's used fewer hardware features.
- Support dropped as of VirtualBox 6.1, which just came out. 6.0 is only
supported until 2020-07.
- Requires some changes to pulplift / box definitions to work

Emulation with Qemu:
Pros:
- Well-Supported
- Has all the other features of qemu-kvm, including emulated CPU count
- Needs only 1 easily-set setting needed for pulplift to work with it
Cons:
- Very slow. Rule of thumb is at best, 10% of native performance.

-Mike

On Thu, Feb 13, 2020 at 6:14 PM Mike DePaulo <mikedep333 at redhat.com> wrote:

> I've only tested Travis so far, but this is very promising.
>
> Hardware KVM virtualization appears to be working on Travis, via pulplift
> (which uses vagrant, libvirt & KVM), without any hacks!
>
> My current theory is that Travis uses either OpenVZ or KVM, and that the
> "svm" warning is a limitation of nested virtualization working properly.
>
> I'm going to investigate Travis a little further before trying out GHA.
> (Or test them in parallel with same commands.) Including making 100% sure
> it is not falling back to unaccelerated qemu emulation.
>
> $ uname -a
> Linux travis-job-7dcf26ac-24c0-462e-8418-69c466817f8e 5.0.0-1026-gcp
> #27~18.04.1-Ubuntu SMP Fri Nov 15 07:40:39 UTC 2019 x86_64 x86_64 x86_64
> GNU/Linux
>
> $ sudo virt-what
> kvm
>
> $ sudo qemu-system-x86_64 -machine accel=kvm -vnc 127.0.0.1:1
> qemu-system-x86_64: warning: host doesn't support requested feature:
> CPUID.80000001H:ECX.svm [bit 2]
> ^C
>
> $ sudo vagrant ssh fedora31
> Last login: Thu Feb 13 22:55:40 2020 from 192.168.121.1
> $ uname -a
> Linux localhost.localdomain 5.3.7-301.fc31.x86_64 #1 SMP Mon Oct 21
> 19:18:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ cat /etc/redhat-release
> Fedora release 31 (Thirty One)
>
> -Mike
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat <https://www.redhat.com/>
>
> IM: mikedep333
>
> GPG: 51745404
> <https://www.redhat.com/>
>


-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat <https://www.redhat.com/>

IM: mikedep333

GPG: 51745404
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200218/6b9c868e/attachment.htm>