<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Hi all,<br>
    <br>
    <big><font color="#663300"><b>Problem:</b></font></big> Nested VM
    images fail/freeze during booting.<br>
    <br>
    <font color="#663300"><big><b>Details:</b></big></font><br>
    <br>
    <b>Hardware:</b><br>
    <ul>
      <li>Model: Dell R620</li>
      <li>CPU: 2 x 6 x 2 (ht) Intel E5-2667</li>
      <ul>
        <li>flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
          mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
          pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon
          pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf
          eagerfpu pni pclmulqdq dtes64 monitor ds_cpl <b>vmx </b>smx
          est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic
          popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat
          xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid<br>
        </li>
      </ul>
      <li>Memory: 192GB</li>
      <li>Disk: 7.0T local filesystem<br>
      </li>
    </ul>
    <b>Physical Hypervisor Configuration:</b><br>
    <ul>
      <li>/etc/modprobe.d/kvm-nested.conf (see below)</li>
      <li>/etc/libvirt/qemu/virtual-hypervisor.xml (see below)</li>
      <ul>
        <li>Runs without any problems</li>
        <li>Includes: <cpu ...><feature policy='require'
          name='vmx'/></cpu><br>
        </li>
      </ul>
    </ul>
    <b>Virtual Hypervisor Configuration:</b><br>
    <ul>
      <li>virt-manager opens without any problems</li>
      <li>The virtual guest within the virtual hypervisor VM fails to
        boot </li>
      <ul>
        <li>I see the screen in which the boot choices are shown.  After
          the selection is made (either by waiting for the selection
          timeout, or by explicit selection), the next screen shows a
          screen dump (see below)<br>
        </li>
      </ul>
      <li>/etc/libvirt/qemu/virtual-guest.xml (see below)</li>
      <ul>
        <li>Uses virbr0</li>
        <li>There is no /var/log/libvirt/qemu logfile as the boot up
          doesn't get far enough.<br>
        </li>
      </ul>
    </ul>
    I tried placing the virtual-guest.xml and image on the physical
    hypervisor and it does boot up without any problems.<br>
    <br>
    I've scoured the web and can't figure out why boot up is failing for
    the nested VM.  Can anyone think of a solution or help out in any
    way?<br>
    <br>
    Thanks,<br>
    Steve Amerige<br>
    <small> Principal Software Developer, Fraud and Compliance Solutions
      Development<br>
      SAS Institute, 100 SAS Campus Dr, Room U3050, Cary, NC 27513-8617
    </small>
    <br>
    <br>
    <big><font color="#663300"><b>Physical Hypervisor Configuration:</b></font></big><br>
    <br>
    <b>/etc/modprobe.d/kvm-nested.conf:</b><br>
    <br>
    <small><tt>options kvm_intel nested=1</tt></small><br>
    <br>
    <b>/etc/libvirt/qemu/virtual-hypervisor.xml:</b><br>
    <small><tt><br>
      </tt><tt><domain type='kvm'></tt><tt><br>
      </tt><tt>  <name>virtual-hypervisor</name></tt><tt><br>
      </tt><tt> 
        <uuid>5bded4c9-e00a-4c33-8d68-f79ef9bf6daa</uuid></tt><tt><br>
      </tt><tt>  <memory unit='KiB'>16777216</memory></tt><tt><br>
      </tt><tt>  <currentMemory
        unit='KiB'>33554432</currentMemory></tt><tt><br>
      </tt><tt>  <vcpu placement='static'>4</vcpu></tt><tt><br>
      </tt><tt>  <os></tt><tt><br>
      </tt><tt>    <type arch='x86_64'
        machine='pc-i440fx-rhel7.0.0'>hvm</type></tt><tt><br>
      </tt><tt>    <boot dev='hd'/></tt><tt><br>
      </tt><tt>  </os></tt><tt><br>
      </tt><tt>  <features></tt><tt><br>
      </tt><tt>    <acpi/></tt><tt><br>
      </tt><tt>    <apic/></tt><tt><br>
      </tt><tt>    <pae/></tt><tt><br>
      </tt><tt>  </features></tt><tt><br>
      </tt><tt>  <cpu mode='custom' match='exact'></tt><tt><br>
      </tt><tt>    <model
        fallback='allow'>SandyBridge</model></tt><tt><br>
      </tt><tt>    <vendor>Intel</vendor></tt><tt><br>
      </tt><tt>    <feature policy='require' name='vmx'/></tt><tt><br>
      </tt><tt>  </cpu></tt><tt><br>
      </tt><tt>  <clock offset='utc'/></tt><tt><br>
      </tt><tt>  <on_poweroff>destroy</on_poweroff></tt><tt><br>
      </tt><tt>  <on_reboot>restart</on_reboot></tt><tt><br>
      </tt><tt>  <on_crash>restart</on_crash></tt><tt><br>
      </tt><tt>  <devices></tt><tt><br>
      </tt><tt>   
        <emulator>/usr/libexec/qemu-kvm</emulator></tt><tt><br>
      </tt><tt>    <disk type='file' device='disk'></tt><tt><br>
      </tt><tt>      <driver name='qemu' type='qcow2'
        cache='none'/></tt><tt><br>
      </tt><tt>      <source
        file='/var/lib/libvirt/images/virtual-hypervisor.qcow2'/></tt><tt><br>
      </tt><tt>      <target dev='vda' bus='virtio'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x05' function='0x0'/></tt><tt><br>
      </tt><tt>    </disk></tt><tt><br>
      </tt><tt>    <disk type='block' device='cdrom'></tt><tt><br>
      </tt><tt>      <driver name='qemu' type='raw'/></tt><tt><br>
      </tt><tt>      <target dev='hdc' bus='ide'/></tt><tt><br>
      </tt><tt>      <readonly/></tt><tt><br>
      </tt><tt>      <address type='drive' controller='0' bus='1'
        target='0' unit='0'/></tt><tt><br>
      </tt><tt>    </disk></tt><tt><br>
      </tt><tt>    <controller type='usb' index='0'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x01' function='0x2'/></tt><tt><br>
      </tt><tt>    </controller></tt><tt><br>
      </tt><tt>    <controller type='pci' index='0'
        model='pci-root'/></tt><tt><br>
      </tt><tt>    <controller type='ide' index='0'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x01' function='0x1'/></tt><tt><br>
      </tt><tt>    </controller></tt><tt><br>
      </tt><tt>    <interface type='bridge'></tt><tt><br>
      </tt><tt>      <mac address='52:54:00:7a:40:a6'/></tt><tt><br>
      </tt><tt>      <source bridge='br0'/></tt><tt><br>
      </tt><tt>      <model type='virtio'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x03' function='0x0'/></tt><tt><br>
      </tt><tt>    </interface></tt><tt><br>
      </tt><tt>    <serial type='pty'></tt><tt><br>
      </tt><tt>      <target port='0'/></tt><tt><br>
      </tt><tt>    </serial></tt><tt><br>
      </tt><tt>    <console type='pty'></tt><tt><br>
      </tt><tt>      <target type='serial' port='0'/></tt><tt><br>
      </tt><tt>    </console></tt><tt><br>
      </tt><tt>    <input type='tablet' bus='usb'/></tt><tt><br>
      </tt><tt>    <input type='mouse' bus='ps2'/></tt><tt><br>
      </tt><tt>    <graphics type='vnc' port='-1' autoport='yes'/></tt><tt><br>
      </tt><tt>    <sound model='ich6'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x04' function='0x0'/></tt><tt><br>
      </tt><tt>    </sound></tt><tt><br>
      </tt><tt>    <video></tt><tt><br>
      </tt><tt>      <model type='cirrus' vram='9216' heads='1'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x02' function='0x0'/></tt><tt><br>
      </tt><tt>    </video></tt><tt><br>
      </tt><tt>    <memballoon model='virtio'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x06' function='0x0'/></tt><tt><br>
      </tt><tt>    </memballoon></tt><tt><br>
      </tt><tt>  </devices></tt><tt><br>
      </tt><tt></domain></tt></small><br>
    <br>
    <font color="#663300"><big><b>Virtual Guest Configuration:</b></big></font><br>
    <br>
    <small><tt><domain type='kvm'></tt><tt><br>
      </tt><tt>  <name>virtual-guest.xml</name></tt><tt><br>
      </tt><tt><uuid>863ffbe3-1d4e-4ca6-b86f-bb13e32fed6b</uuid></tt><tt><br>
      </tt><tt>  <memory unit='KiB'>2097152</memory></tt><tt><br>
      </tt><tt>  <currentMemory
        unit='KiB'>2097152</currentMemory></tt><tt><br>
      </tt><tt>  <vcpu placement='static'>1</vcpu></tt><tt><br>
      </tt><tt>  <os></tt><tt><br>
      </tt><tt>    <type arch='x86_64'
        machine='pc-i440fx-rhel7.0.0'>hvm</type></tt><tt><br>
      </tt><tt>    <boot dev='hd'/></tt><tt><br>
      </tt><tt>  </os></tt><tt><br>
      </tt><tt>  <features></tt><tt><br>
      </tt><tt>    <acpi/></tt><tt><br>
      </tt><tt>    <apic/></tt><tt><br>
      </tt><tt>    <pae/></tt><tt><br>
      </tt><tt>  </features></tt><tt><br>
      </tt><tt>  <clock offset='utc'/></tt><tt><br>
      </tt><tt>  <on_poweroff>destroy</on_poweroff></tt><tt><br>
      </tt><tt>  <on_reboot>restart</on_reboot></tt><tt><br>
      </tt><tt>  <on_crash>restart</on_crash></tt><tt><br>
      </tt><tt>  <devices></tt><tt><br>
      </tt><tt>   
        <emulator>/usr/libexec/qemu-kvm</emulator></tt><tt><br>
      </tt><tt>    <disk type='file' device='disk'></tt><tt><br>
      </tt><tt>      <driver name='qemu' type='qcow2'
        cache='none'/></tt><tt><br>
      </tt><tt>      <source
        file='/var/lib/libvirt/images/virtual-guest.xml.img'/></tt><tt><br>
      </tt><tt>      <target dev='vda' bus='virtio'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x05' function='0x0'/></tt><tt><br>
      </tt><tt>    </disk></tt><tt><br>
      </tt><tt>    <disk type='block' device='cdrom'></tt><tt><br>
      </tt><tt>      <driver name='qemu' type='raw'/></tt><tt><br>
      </tt><tt>      <target dev='hdc' bus='ide'/></tt><tt><br>
      </tt><tt>      <readonly/></tt><tt><br>
      </tt><tt>      <address type='drive' controller='0' bus='1'
        target='0' unit='0'/></tt><tt><br>
      </tt><tt>    </disk></tt><tt><br>
      </tt><tt>    <controller type='usb' index='0'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x01' function='0x2'/></tt><tt><br>
      </tt><tt>    </controller></tt><tt><br>
      </tt><tt>    <controller type='pci' index='0'
        model='pci-root'/></tt><tt><br>
      </tt><tt>    <controller type='ide' index='0'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x01' function='0x1'/></tt><tt><br>
      </tt><tt>    </controller></tt><tt><br>
      </tt><tt>    <interface type='bridge'></tt><tt><br>
      </tt><tt>    <mac address='52:54:00:0f:a8:81'/></tt><tt><br>
      </tt><tt>      <source bridge='virbr0'/></tt><tt><br>
      </tt><tt>      <model type='virtio'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x03' function='0x0'/></tt><tt><br>
      </tt><tt>    </interface></tt><tt><br>
      </tt><tt>    <serial type='pty'></tt><tt><br>
      </tt><tt>      <target port='0'/></tt><tt><br>
      </tt><tt>    </serial></tt><tt><br>
      </tt><tt>    <console type='pty'></tt><tt><br>
      </tt><tt>      <target type='serial' port='0'/></tt><tt><br>
      </tt><tt>    </console></tt><tt><br>
      </tt><tt>    <input type='tablet' bus='usb'/></tt><tt><br>
      </tt><tt>    <input type='mouse' bus='ps2'/></tt><tt><br>
      </tt><tt>    <graphics type='vnc' port='-1' autoport='yes'/></tt><tt><br>
      </tt><tt>    <sound model='ich6'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x04' function='0x0'/></tt><tt><br>
      </tt><tt>    </sound></tt><tt><br>
      </tt><tt>    <video></tt><tt><br>
      </tt><tt>      <model type='cirrus' vram='9216' heads='1'/></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x02' function='0x0'/></tt><tt><br>
      </tt><tt>    </video></tt><tt><br>
      </tt><tt>    <memballoon model='virtio'></tt><tt><br>
      </tt><tt>      <address type='pci' domain='0x0000' bus='0x00'
        slot='0x06' function='0x0'/></tt><tt><br>
      </tt><tt>    </memballoon></tt><tt><br>
      </tt><tt>  </devices></tt><tt><br>
      </tt><tt></domain></tt></small><br>
    <br>
    <font color="#663300"><big><b>virt-manager display of Virtual
          Machine showing failure:<br>
          <br>
          <img alt="" src="cid:part1.05040805.01040208@sas.com"
            height="635" width="857"><br>
        </b></big></font>
  </body>
</html>