[Libguestfs] Double fault panic in L2 upon v2v conversion

Thu Jan 16 05:43:44 UTC 2014

There is a bug which is still not fixed:Bug 956580 -[ RFE ]Support for VMware & Citrix OVF image import into RHEV
https://bugzilla.redhat.com/show_bug.cgi?id=956580

Best regards,
Tingting Zheng（郑婷婷）

----- Original Message -----
From: "Rom Freiman" <rom at stratoscale.com>
To: libguestfs at redhat.com
Sent: Wednesday, January 15, 2014 10:35:29 PM
Subject: [Libguestfs] Double fault panic in L2 upon v2v conversion

Hi everybody, 

Wanted to hear your opinion and to receive a smart advice. 

I'm trying to use virt-v2v in order to convert ova image (exported from vcenter) to run on libvirt/kvm - all this inside a VM of fedora. 
The converted image is also a fedora. 
During the conversion process, in some point of libguestfs activity, I get double fault panic from L2 (printed as part of libguest output) and the conversion process fails - no errors appear neither in L0 not in L1 message logs. 

This is more detailed explanation: 

Description of problem:
During v2v conversion for ova fedora image in a VM there is double fault panic happens and L2 crashes during libguestfs conversion.

The crash happens in different stages, but usually upon memory pressure in L0.

There are no error logs in L1 and cannot find strong correlation to patches that were added to L0-KVM to avoid L0 crash upon nested vm with high memory pressure - http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9 .

Command within L1 to perform the conversion (fedora.ova - vmdk image of fedora - was placed in advance on the VM):

LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default -oc qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova

Version-Release number of selected component (if applicable):
L0: 
Kernel: 3.11.8-200.fc19 + nested crash patches
libvritd: 1.0.5.8
qemu: 1.6.1
libguestfs-test-tool 1.22.7fedora=19,release=1.fc19,libvirt
L1:
Kernel: 3.11.8-200..fc19.x86_64
libvirtd: 1.0.5.8
qemu: 1.6.1 + v2v patch (skip vmdk version verification)
libguestfs-test-tool 1.22.7fedora=19,release=2.fc19,libvirt
virt-v2v 0.9.0

L2:
Kernel: 3.11.10-301.fc20

How reproducible:
LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default -oc qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova

Steps to Reproduce:
1. Upload ova image to VM
2. Run v2v to perform the conversion
3. Add some memory pressure on L0 (dd if=/dev/urandom of=/tmp/bigfile count=6M)

Actual results: 
libguestfs: checksum of existing appliance: 63a0a4c8acb7f789c7a725bebbb16e64a3bc0999fdca61931babda50292d83b8
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: [00337ms] begin testing qemu features
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.6
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00661ms] finished testing qemu features
[00663ms] /usr/bin/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -nodefconfig \
    -nodefaults \
    -nographic \
    -machine accel=kvm:tcg \
    -m 500 \
    -no-reboot \
    -no-hpet \
    -kernel /var/tmp/.guestfs-0/kernel.1416 \
    -initrd /var/tmp/.guestfs-0/initrd.1416 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/var/lib/libvirt/images/fedora-v2v-disk1.vmdk,cache=none,format=qcow2,id=hd0,if=ide \
    -drive file=/tmp/3pEC1jAXQs,snapshot=on,format=raw,id=hd1,if=ide \
    -drive file=/var/tmp/.guestfs-0/root.1416,snapshot=on,id=appliance,if=none,cache=unsafe \
    -device scsi-hd,drive=appliance \
    -device virtio-serial \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfsKr0x6j/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net= 169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0 guestfs_verbose=1 TERM=dumb'
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 02/18/13
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Mon Feb 18 08:29:02 UTC 2013
Term: 80x24
4 0
SeaBIOS (version ?-20131118_112708-)

iPXE ( http://ipxe.org ) 00:04.0 C100 PCI2.10 PnP PMM+1F3C7C60+1F327C60 C100

Booting from ROM...
Probing EDD (edd=off to disable)... ok
\x1b[2J[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 (mockbuild at ellaria) (gcc version 4.8.2 20131017 (Red Hat 4.8.2-1) (GCC) ) #1 SMP Mon Nov 18 17:54:34 IST 2013
[    0.000000] Command line: panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0 guestfs_verbose=1 TERM=dumb
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable 
............................................ 
.........More logs here............ 
............................................. 
guestfsd: main_loop: new request, len 0x3c
mount -o ro /dev/sdb /sysroot/
[   12.645305] PANIC: double fault, error_code: 0x0
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305] task: ffff88001cc816e0 ti: ffff88001cde6000 task.ti: ffff88001cde6000
[   12.645305] RIP: 0033:[<00007fa602c5b99b>]  [<00007fa602c5b99b>] 0x7fa602c5b99a
[   12.645305] RSP: 002b:00007fff4f5884a0  EFLAGS: 00010216
[   12.645305] RAX: 00007fa602008ff8 RBX: 00007fa601ff0000 RCX: 00007fa601ff0000
[   12.645305] RDX: 00000000003b7068 RSI: 00007fff4f588560 RDI: 00007fa601ff3d18
[   12.645305] RBP: 00007fff4f5885d0 R08: 00007fa60200f310 R09: 0000000000000000
[   12.645305] R10: 0000000000000022 R11: 00007fa60200f310 R12: 00007fa60200e9b0
[   12.645305] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fa602e6e990
[   12.645305] FS:  00007fa602e69880(0000) GS:ffff88001f000000(0000) knlGS:0000000000000000
[   12.645305] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   12.645305] CR2: 0000000000000000 CR3: 000000001d7fb000 CR4: 00000000000006f0
[   12.645305] 
[   12.645305] Kernel panic - not syncing: Machine halted.
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305]  ffff88001f005f58 ffff88001f005e90 ffffffff8164024b ffffffff819e89dc
[   12.645305]  ffff88001f005f08 ffffffff8163c272 0000000000000008 ffff88001f005f18
[   12.645305]  ffff88001f005eb8 ffffffff8163c8e5 0000000000000046 00000000000000b1
[   12.645305] Call Trace:
[   12.645305]  <#DF>  [<ffffffff8164024b>] dump_stack+0x45/0x56
[   12.645305]  [<ffffffff8163c272>] panic+0xc8/0x1d7
[   12.645305]  [<ffffffff8163c8e5>] ? printk+0x67/0x69
[   12.645305]  [<ffffffff81048ae1>] df_debug+0x31/0x40
[   12.645305]  [<ffffffff810132ed>] do_double_fault+0x5d/0x80
[   12.645305]  [<ffffffff81650b88>] double_fault+0x28/0x30
[   12.645305]  <<EOE>> 
[   12.645305] Rebooting in 1 seconds..libguestfs: child_cleanup: 0x3a05f50: child process died
libguestfs: sending SIGTERM to process 1526
libguestfs: trace: mount_ro = -1 (error)
libguestfs: trace: vfs_type "/dev/sda1"
libguestfs: trace: vfs_type = NULL (error)
libguestfs: check_for_filesystem_on: /dev/sda1 (failed to get vfs type)
libguestfs: trace: internal_parse_mountable "/dev/sda1"
libguestfs: trace: internal_parse_mountable = NULL (error)
libguestfs: trace: inspect_os = NULL (error)
libguestfs: trace: close

Additional info:
The same crash also happens when L0 is 3.11.9 (with kvm patch to avoid L0 crash in nested environment - http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9 ) but more rarely. 
Thanks a lot, 
Rom 

_______________________________________________
Libguestfs mailing list
Libguestfs at redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs