[Libguestfs] virt-p2v VM testing bitrot?

[Laszlo Ersek]

On 08/24/22 16:13, Richard W.M. Jones wrote:
> On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
>> I've started looking into RHBZ 1590721 for virt-p2v.
>>
>> For p2v development, quick local testing is helpful. "make
>> run-virt-p2v-directly" seems to be working great; however, the VM-based
>> test methods seem to have developed problems, since I last looked.
>>
>> Namely:
>>
>> - "make run-virt-p2v-in-a-vm" boots quickly, but the GUI does not come up.
>>
>> - "make run-virt-p2v-in-an-nvme-vm" boots *incredibly slowly*. I
>> checked, and the host CPU utilization during guest boot was around 20%
>> (and KVM was enabled). I don't understand how or why, but exposing the
>> "physical machine" disk over NVMe slows guest boot to a crawl -- it
>> looks strangely "IO-bound". I don't recall this from the time I added
>> this Makefile target!
>>
>> Do these symptoms look familiar?
> 
> Yes I had noticed this.  I did look at it briefly but couldn't work
> out what was going wrong.

Thanks for confirming!

I think I've found something, at least in relation to the first issue
(i.e., run-virt-p2v-in-a-vm booting quickly, but not launching the p2v
service). The problem seems to be the following:

- Both the "physical machine" disk (fedora.img) and the bootable p2v
disk (virt-p2v.img) are based on Fedora 35. The former because of commit
3b59d4acb82f ("Use fedora-35 as the test physical machine", 2022-05-10),
the latter because my host system runs Fedora 35, and so
virt-p2v-make-disk picks F35.

- Consequently, both the "hd0" drive and the "usb0" drive are based on
the same Fedora 35 virt-builder template.

- The grub command line (part of *both* images) specifies the *same*
kernel parameter "root=UUID=fae2...".

- This filesystem UUID exists *twice* in the virtual machine however,
once in "hd0" and another time in "usb0" -- duplicate UUID.

- The guest kernel ends up mounting / loading the root filesystem from
"hd0", that is, the "physical machine" Fedora 35 image, which does not
have anything injected that's related to p2v.

I found this by attempting to log in to the guest. The "p2v" password
did not work (which is hardcoded in "virt-p2v-make-disk") , but the
password printed by virt-builder under the PHYSICAL_MACHINE target in
"Makefile" did work. Then, out of curiosity, I tried mounting the
"other" disk in the running VM, but the XFS driver rejected the mount
attempt, due to duplicate UUIDs.

Now, virt-sysprep (and therefore virt-builder) has an operation called
"fs-uuids", but it is not enabled by default:
<https://libguestfs.org/virt-sysprep.1.html#notes-on-fs-uuids>, and we
also don't enable it manually for either "fedora.img" or "virt-p2v.img".

I guess the issue was "masked" before commit 3b59d4acb82f ("Use
fedora-35 as the test physical machine", 2022-05-10), at least on my
workstation. I think the Fedora versions I've used on this workstation
have been F34 and F35, so for "virt-p2v.img", virt-builder would
download F34 and F35 templates, which would differ from the template
used for "fedora.img" (F32) -- hence the ultimately different fs UUIDs.

... I have no news regarding the NVMe slowness, but the above UUID
collision exists in "run-virt-p2v-in-an-nvme-vm" just the same. Once we
fix the UUID conflict, the NVMe test might go back to normal too.

Laszlo


> 
> Rich.
> 
>> (For RHBZ 1590721, I think run-virt-p2v-directly will suffice; just
>> wanted to record the above somewhere.)
>>
>> Thanks
>> Laszlo
>