[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [libvirt-users] Help? Running into problems with migrateToURI2() and virDomainDefCheckABIStability()



On 02/11/2014 04:45 PM, Cole Robinson wrote:
> On 02/10/2014 06:46 PM, Chris Friesen wrote:
>> Hi,
>>
>> We've run into a problem with libvirt 1.1.2 and are looking for some comments
>> on whether this is a bug or design intent.
>>
>> We're trying to use migrateToURI() but we're using a few things (numatune,
>> vcpu mask, etc.) that may need adjustment during the migration.  We found that
>> migrateToURI2() mostly works if we use XML created by copying the domain XML
>> from the running instance and modifying the appropriate sections.
>>
>> The problem that we're seeing is that the serial console checking in
>> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing
>> even though we haven't touched the serial console XML:

> I think we will have to see the XML your passing before and after edit to
> figure out what's going wrong. Also, showing all the arguments you are
> invoking the migration APIs with.


Okay, this is going to be a long message.  :)

We're calling it as:

dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth)

Here are the supplied arguments:
duri = u'qemu+tcp://compute-0/system'
muri = u'tcp://compute-0'
flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE']
logical_sum = 19,  (logical OR of flaglist)
CONF.live_migration_bandwidth = 0

Grabbing original XML prior to migration:

dom_xml = etree.fromstring(dom.XMLDesc(0))
xml_str = etree.tostring(dom_xml)

xml_str (BEFORE) =
<domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1-2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="2"/>
    <emulatorpin cpuset="1-2"/>
  </cputune>
  <numatune>
    <memory mode="preferred" nodeset="0"/>
  </numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216" heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>



This is the modified XML, specifically updating vcpu, cputune, numatune.
In this case they actually have the same values as above but there are
whitespace differences:


xml_str (MODIFIED) =
<domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1,2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/><vcpupin vcpu="1" cpuset="2"/><emulatorpin cpuset="1,2"/></cputune>
  <numatune>
    <memory mode="preferred" nodeset="0"/></numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216" heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>



Calling migrateToURI2 then gives the following error :
2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance: 2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0


If we remove the this section :
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>

Then we get the following error :
2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance: 90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration: Target domain serial port count 1 does not match source 2

If we remove just the following line:
      <source path="/dev/pts/1"/>

so the modified portion looks like this,
    <serial type="file">
      <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">

Then we get this error :

2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance: 00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0

If we modify it with empty <serial type='pty'> element like this:

    <serial type="file">
      <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      </serial>
    <console type="file">
      <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>

Then we get this error:

2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance: b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0

If we modify it with empty <serial type='pty'> element and also remove
<console type='file'> section completely like this :

    <serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>

Then this successfully migrates!

However, it then fails on subsequent migration back to the original host.  Prior to migrating
back the domain XML serial/console section looks like:

    <serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>

After modifying empty <serial type='pty'> element and removing the <console type='file'> section
it looks like this :

    <serial type="file">
      <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>

But this give the following error when migrating back:

2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance: 8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration: Target domain console count 0 does not match source 1



Chris


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]