[libvirt] [libvirt-users] Help? Running into problems with migrateToURI2() and virDomainDefCheckABIStability()

Cole Robinson crobinso at redhat.com
Wed Feb 12 18:35:02 UTC 2014


On 02/12/2014 01:03 PM, Chris Friesen wrote:
> On 02/11/2014 04:45 PM, Cole Robinson wrote:
>> On 02/10/2014 06:46 PM, Chris Friesen wrote:
>>> Hi,
>>>
>>> We've run into a problem with libvirt 1.1.2 and are looking for some comments
>>> on whether this is a bug or design intent.
>>>
>>> We're trying to use migrateToURI() but we're using a few things (numatune,
>>> vcpu mask, etc.) that may need adjustment during the migration.  We found that
>>> migrateToURI2() mostly works if we use XML created by copying the domain XML
>>> from the running instance and modifying the appropriate sections.
>>>
>>> The problem that we're seeing is that the serial console checking in
>>> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing
>>> even though we haven't touched the serial console XML:
> 
>> I think we will have to see the XML your passing before and after edit to
>> figure out what's going wrong. Also, showing all the arguments you are
>> invoking the migration APIs with.
> 
> 
> Okay, this is going to be a long message.  :)
> 
> We're calling it as:
> 
> dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth)
> 
> Here are the supplied arguments:
> duri = u'qemu+tcp://compute-0/system'
> muri = u'tcp://compute-0'
> flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE']
> logical_sum = 19,  (logical OR of flaglist)
> CONF.live_migration_bandwidth = 0
> 
> Grabbing original XML prior to migration:
> 
> dom_xml = etree.fromstring(dom.XMLDesc(0))
> xml_str = etree.tostring(dom_xml)
> 
> xml_str (BEFORE) =
> <domain type="qemu" id="2">
>   <name>instance-00000006</name>
>   <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
>   <dpdk>
>     <process type="secondary"/>
>     <file prefix="vs"/>
>     <cpu list="0"/>
>     <memory channels="4"/>
>   </dpdk>
>   <memory unit="KiB">262144</memory>
>   <currentMemory unit="KiB">262144</currentMemory>
>   <memoryBacking>
>     <hugepages/>
>   </memoryBacking>
>   <vcpu placement="static" cpuset="1-2">2</vcpu>
>   <cputune>
>     <vcpupin vcpu="0" cpuset="1"/>
>     <vcpupin vcpu="1" cpuset="2"/>
>     <emulatorpin cpuset="1-2"/>
>   </cputune>
>   <numatune>
>     <memory mode="preferred" nodeset="0"/>
>   </numatune>
>   <sysinfo type="smbios">
>     <system>
>       <entry name="manufacturer">OpenStack Foundation</entry>
>       <entry name="product">OpenStack Nova</entry>
>       <entry name="version">2013.2.2</entry>
>       <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
>       <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
>     </system>
>   </sysinfo>
>   <os>
>     <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
>     <boot dev="hd"/>
>     <smbios mode="sysinfo"/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>   </features>
>   <clock offset="utc"/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/bin/qemu-system-x86_64</emulator>
>     <disk type="file" device="disk">
>       <driver name="qemu" type="qcow2" cache="none"/>
>       <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
>       <target dev="vda" bus="virtio"/>
>       <alias name="virtio-disk0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
>     </disk>
>     <controller type="usb" index="0">
>       <alias name="usb0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
>     </controller>
>     <controller type="pci" index="0" model="pci-root">
>       <alias name="pci.0"/>
>     </controller>
>     <interface type="vswitch">
>       <mac address="fa:16:3e:fb:34:6b"/>
>       <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
>       <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
>       <model type="e1000"/>
>       <alias name="net0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
>     </interface>
>     <interface type="vswitch">
>       <mac address="fa:16:3e:d5:b9:50"/>
>       <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
>       <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
>       <model type="e1000"/>
>       <alias name="net1"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
>     </interface>
>     <serial type="file">
>       <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>       <source path="/dev/pts/1"/>
>       <target port="1"/>
>       <alias name="serial1"/>
>     </serial>
>     <console type="file">
>       <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
>       <target type="serial" port="0"/>
>       <alias name="serial0"/>
>     </console>
>     <input type="tablet" bus="usb">
>       <alias name="input0"/>
>     </input>
>     <input type="mouse" bus="ps2"/>
>     <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
>       <listen type="address" address="0.0.0.0"/>
>     </graphics>
>     <video>
>       <model type="cirrus" vram="9216" heads="1"/>
>       <alias name="video0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
>     </video>
>     <memballoon model="virtio">
>       <alias name="balloon0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
>     </memballoon>
>   </devices>
>   <seclabel type="none"/>
> </domain>
> 
> 
> 
> This is the modified XML, specifically updating vcpu, cputune, numatune.
> In this case they actually have the same values as above but there are
> whitespace differences:
> 
> 
> xml_str (MODIFIED) =
> <domain type="qemu" id="2">
>   <name>instance-00000006</name>
>   <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
>   <dpdk>
>     <process type="secondary"/>
>     <file prefix="vs"/>
>     <cpu list="0"/>
>     <memory channels="4"/>
>   </dpdk>
>   <memory unit="KiB">262144</memory>
>   <currentMemory unit="KiB">262144</currentMemory>
>   <memoryBacking>
>     <hugepages/>
>   </memoryBacking>
>   <vcpu placement="static" cpuset="1,2">2</vcpu>
>   <cputune>
>     <vcpupin vcpu="0" cpuset="1"/><vcpupin vcpu="1" cpuset="2"/><emulatorpin cpuset="1,2"/></cputune>
>   <numatune>
>     <memory mode="preferred" nodeset="0"/></numatune>
>   <sysinfo type="smbios">
>     <system>
>       <entry name="manufacturer">OpenStack Foundation</entry>
>       <entry name="product">OpenStack Nova</entry>
>       <entry name="version">2013.2.2</entry>
>       <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
>       <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
>     </system>
>   </sysinfo>
>   <os>
>     <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type>
>     <boot dev="hd"/>
>     <smbios mode="sysinfo"/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>   </features>
>   <clock offset="utc"/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/bin/qemu-system-x86_64</emulator>
>     <disk type="file" device="disk">
>       <driver name="qemu" type="qcow2" cache="none"/>
>       <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
>       <target dev="vda" bus="virtio"/>
>       <alias name="virtio-disk0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
>     </disk>
>     <controller type="usb" index="0">
>       <alias name="usb0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
>     </controller>
>     <controller type="pci" index="0" model="pci-root">
>       <alias name="pci.0"/>
>     </controller>
>     <interface type="vswitch">
>       <mac address="fa:16:3e:fb:34:6b"/>
>       <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
>       <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
>       <model type="e1000"/>
>       <alias name="net0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
>     </interface>
>     <interface type="vswitch">
>       <mac address="fa:16:3e:d5:b9:50"/>
>       <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
>       <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
>       <model type="e1000"/>
>       <alias name="net1"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
>     </interface>
>     <serial type="file">
>       <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>       <source path="/dev/pts/1"/>
>       <target port="1"/>
>       <alias name="serial1"/>
>     </serial>
>     <console type="file">
>       <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
>       <target type="serial" port="0"/>
>       <alias name="serial0"/>
>     </console>
>     <input type="tablet" bus="usb">
>       <alias name="input0"/>
>     </input>
>     <input type="mouse" bus="ps2"/>
>     <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us">
>       <listen type="address" address="0.0.0.0"/>
>     </graphics>
>     <video>
>       <model type="cirrus" vram="9216" heads="1"/>
>       <alias name="video0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
>     </video>
>     <memballoon model="virtio">
>       <alias name="balloon0"/>
>       <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
>     </memballoon>
>   </devices>
>   <seclabel type="none"/>
> </domain>
> 
> 
> 
> Calling migrateToURI2 then gives the following error :
> 2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance: 2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
> 
> 
> If we remove the this section :
>     <serial type="pty">
>       <source path="/dev/pts/1"/>
>       <target port="1"/>
>       <alias name="serial1"/>
>     </serial>
> 
> Then we get the following error :
> 2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance: 90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration: Target domain serial port count 1 does not match source 2
> 
> If we remove just the following line:
>       <source path="/dev/pts/1"/>
> 
> so the modified portion looks like this,
>     <serial type="file">
>       <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>       <target port="1"/>
>       <alias name="serial1"/>
>     </serial>
>     <console type="file">
>       <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
>       <target type="serial" port="0"/>
>       <alias name="serial0"/>
>     </console>
>     <input type="tablet" bus="usb">
> 
> Then we get this error :
> 
> 2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance: 00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
> 
> If we modify it with empty <serial type='pty'> element like this:
> 
>     <serial type="file">
>       <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>       </serial>
>     <console type="file">
>       <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
>       <target type="serial" port="0"/>
>       <alias name="serial0"/>
>     </console>
> 
> Then we get this error:
> 
> 2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance: b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0
> 
> If we modify it with empty <serial type='pty'> element and also remove
> <console type='file'> section completely like this :
> 
>     <serial type="file">
>       <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>     </serial>
> 
> Then this successfully migrates!
> 
> However, it then fails on subsequent migration back to the original host.  Prior to migrating
> back the domain XML serial/console section looks like:
> 
>     <serial type="file">
>       <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>       <source path="/dev/pts/1"/>
>       <target port="1"/>
>       <alias name="serial1"/>
>     </serial>
>     <console type="file">
>       <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
>       <target type="serial" port="0"/>
>       <alias name="serial0"/>
>     </console>
> 
> After modifying empty <serial type='pty'> element and removing the <console type='file'> section
> it looks like this :
> 
>     <serial type="file">
>       <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
>       <target port="0"/>
>       <alias name="serial0"/>
>     </serial>
>     <serial type="pty">
>     </serial>
> 
> But this give the following error when migrating back:
> 
> 2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance: 8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration: Target domain console count 0 does not match source 1
> 

Thanks for the detailed info. Can you put that into an upstream libvirt bug
report? http://libvirt.org/bugs.html

Does this sound familiar to any other libvirt devs?

- Cole




More information about the libvir-list mailing list