[Libguestfs] virt-resize - guest unbootable after update-initramfs

Fri Jun 26 13:25:58 UTC 2020

Hi,

I am having a hard time rationalizing why virt-resize would be the
cause of this issue, however it appears to definitely be the trigger,
hence trying this list.

My environment comprises Debian 10 hosts and guests, although this
issue has also been witnessed on Debian 9 hosts running Debian9/10
guests. In all cases, stable versions of libvirt, qemu-kvm, libguestfs-
tools, etc running on AMD64 arch.

The issue is that for guests whose disks have been populated by virt-
resize, when they first run `update-initramfs` (e.g. after a kernel
update), the system will become unbootable. In this state it fails to
the initramfs debug shell where it can be seen that none of the IO
modules have loaded, and so it is unable to mount root:

"mount: mounting /dev/vda1 on /root failed: No such device"

My workflow is as follows:

* Prereq: I have a 'base' guest, backed by a small LVM volume
containing / & swap (xfs, but also tried ext4). This is a vanilla
Debian 10 install, with some very minor mods (a couple of packages
added, a user created, IPV6 disabled)
* For a new guest, I then have a script:
  * Creates an LVM snapshot of the base guest volume
  * `kpartx -a /path/to/snapshot` to access the partitions
  * mount / and then small edits to /etc/hostname and
/etc/network/interfaces to configure networking
  * `kpartx -d /path/to/snapshot`
  * Create a new LVM volume at the size required for the final guest
  * `virt-resize` to copy the partitions from the snapshot to the new
volume, resizing swap to a fixed size and then growing / to fill the
remainder
  * Generate an XML config file to define the new host, based on the
XML used to create the base host (making the hosts identical, other
than their unqiue differences, UUID, MAC, disks, etc)

At this point the guest will boot OK, and continue to run without
issue, including any number of reboots, until the first time update-
initramfs is run. At this point the guest fails to boot.

By contrast, similar guests that have disks that have not been mananged
by `virt-resize` do not experience this issue. I have performed a wide
array of swap tests between source volumes, snapshots, volumes created
by a `dd` copy and `virt-resize` each connected to different hosts,
including the original base host and those derived from it. The only
common thread so far, is the problem occurs with disks populated by
`virt-resize`. 

Booting a guest pre/post having update-initramfs run and passing
"break=premount" to the kernel to gain access to the initramfs debug
shell and then checking the output of `dmesg`, show identical output,
right up until the end when you see the following differences:

Host prior to update-initramfs:

"
[    0.736291] Run /init as init process
[    0.795060] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized
[    0.800259] PCI Interrupt Link [GSIA] enabled at IRQ 16
[    0.800363] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[    0.811428] cryptd: max_cpu_qlen set to 1000
[    0.821407] AVX2 version of gcm_enc/dec engaged.
[    0.821407] AES CTR mode by8 optimization enabled
[    0.823702] ACPI: bus type USB registered
[    0.823712] usbcore: registered new interface driver usbfs
[    0.823717] usbcore: registered new interface driver hub
[    0.823726] usbcore: registered new device driver usb
[    0.829766] SCSI subsystem initialized
[    0.830240] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input3
[    0.830430] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input2
[    0.844882] virtio_blk virtio2: [vda] 41943040 512-byte logical
blocks (21.5 GB/20.0 GiB)
[    0.854231]  vda: vda1 vda2
[    0.857633] xhci_hcd 0000:02:00.0: xHCI Host Controller
[    0.857638] xhci_hcd 0000:02:00.0: new USB bus registered, assigned
bus number 1
[    0.857900] xhci_hcd 0000:02:00.0: hcc params 0x00087001 hci version
0x100 quirks 0x0000000000000010
[    0.859088] virtio_net virtio0 enp1s0: renamed from eth0
[    0.860055] usb usb1: New USB device found, idVendor=1d6b,
idProduct=0002, bcdDevice= 4.19
[    0.860056] usb usb1: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[    0.860056] usb usb1: Product: xHCI Host Controller
[    0.860057] usb usb1: Manufacturer: Linux 4.19.0-9-amd64 xhci-hcd
[    0.860057] usb usb1: SerialNumber: 0000:02:00.0
"

Host after to update-initramfs:

"
[    0.743940] Run /init as init process
[    0.801833] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized
[    0.814263] cryptd: max_cpu_qlen set to 1000
[    0.817594] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input3
[    0.817785] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input2
[    0.820087] PCI Interrupt Link [GSIA] enabled at IRQ 16
[    0.820187] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[    0.829008] AVX2 version of gcm_enc/dec engaged.
[    0.829008] AES CTR mode by8 optimization enabled
[    0.839844] virtio_blk virtio2: [vda] 41943040 512-byte logical
blocks (21.5 GB/20.0 GiB)
[    0.843734]  vda: vda1 vda2
[    0.846319] virtio_net virtio0 enp1s0: renamed from eth0
"

(I have trimmed the output in the case of the host prior as it just
goes on to load all the expected modules, however in the after case,
that is the end of the dmesg output)

Does anybody have any suggestions as to what might cause virt-resize to
directly/indirectly trigger this condition?

I can supply any necessary diag information or run any tests, but don't
want to make this email even longer with redundant/distracting info.

Many thanks in advance,
Stewart