[libvirt-users] libvirt unavailable while a VM is in migration?

Igor Serebryany igor47 at moomers.org
Mon Dec 20 10:06:37 UTC 2010 

On Mon, Dec 20, 2010 at 10:37:56AM +0100, Jiri Denemark wrote:
> in your first email, you said you can't even run virsh list while migration is
> running, right?

yup -- when i start my migration using my python application, 'virsh
list' no longer works from the command-line. i will try setting up a
simple test-case i can use with 'virsh migrate' and let you know the
results.

> So could you make sure you have debug info libvirt packages installed
> or in case you compile libvirt yourself that you compile with -g
> (should be there by default) and don't strip binaries.

I am using the rpm packages distributed from the website, which I've
installed on my debian box using alien.

I've attached two gdb traces -- one for when a migration is just
running, and another where I've also got a blocked 'virsh list' from the
command line. It appears there's no difference between them (except in
the data counts in the thread actually doing the migration).

I've also attached a trace for the 'virsh' process itself.

Thanks for answering, hopefully we can get to the bottom of this.

--Igor
-------------- next part --------------

Thread 7 (Thread 0x7f6358f33710 (LWP 24325)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00007f635cb560e9 in _L_lock_953 () from /lib/libpthread.so.0
#2  0x00007f635cb55f0b in __pthread_mutex_lock (mutex=0x7f6350015910) at pthread_mutex_lock.c:61
#3  0x000000000043d44d in qemuDriverLock (conn=0x7f635009b8f0) at qemu/qemu_driver.c:144
#4  qemudClose (conn=0x7f635009b8f0) at qemu/qemu_driver.c:4500
#5  0x00007f636054a97b in virReleaseConnect (conn=0x7f635009b8f0) at datatypes.c:269
#6  0x00007f636054b028 in virUnrefConnect (conn=0x7f635009b8f0) at datatypes.c:321
#7  0x00007f63605525c8 in virConnectClose (conn=0x7f635009b8f0) at libvirt.c:1548
#8  0x000000000041941f in qemudFreeClient (opaque=0xc636f0) at libvirtd.c:2284
#9  qemudRunLoop (opaque=0xc636f0) at libvirtd.c:2358
#10 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f634effd710 (LWP 24476)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ab0) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
Current language:  auto
The current source language is "auto; currently asm".

Thread 5 (Thread 0x7f634e7fc710 (LWP 24478)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ac8) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f634dffb710 (LWP 24489)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ae0) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f634d7fa710 (LWP 24494)):
#0  0x00007f635cb5b0bd in read () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f636051683e in read (fd=54, buf=0x13bddb0, count=32768) at /usr/include/bits/unistd.h:45
#2  saferead (fd=54, buf=0x13bddb0, count=32768) at util/util.c:95
#3  0x0000000000447e03 in doTunnelSendAll (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, 
    dom_xml=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, 
    resource=0, dom=<value optimized out>) at qemu/qemu_driver.c:11389
#4  doTunnelMigrate (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, dom_xml=<value optimized out>, 
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0, dom=<value optimized out>)
    at qemu/qemu_driver.c:11604
#5  0x0000000000448631 in doPeer2PeerMigrate (dom=0xc63970, cookie=<value optimized out>, 
    cookielen=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, 
    resource=0) at qemu/qemu_driver.c:11746
#6  qemudDomainMigratePerform (dom=0xc63970, cookie=<value optimized out>, cookielen=<value optimized out>, 
---Type <return> to continue, or q <return> to quit---
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0)
    at qemu/qemu_driver.c:11816
#7  0x00007f6360555b16 in virDomainMigratePerform (domain=0xc63970, cookie=0x0, cookielen=0, 
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, bandwidth=0) at libvirt.c:3931
#8  0x0000000000425782 in remoteDispatchDomainMigratePerform (server=<value optimized out>, 
    client=<value optimized out>, conn=0x7f63501472a0, hdr=<value optimized out>, rerr=0x7f634d7f9be0, 
    args=0x7f634d7f9d20, ret=0x7f634d7f9cc0) at remote.c:1852
#9  0x000000000042b417 in remoteDispatchClientCall (server=0xc636f0, client=0xc66fc0, msg=0xf6d650)
    at dispatch.c:530
#10 remoteDispatchClientRequest (server=0xc636f0, client=0xc66fc0, msg=0xf6d650) at dispatch.c:408
#11 0x000000000041aaa8 in qemudWorker (data=0xc74af8) at libvirtd.c:1586
#12 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#13 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f635672e710 (LWP 24853)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74a68) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f6360ce07e0 (LWP 24324)):
#0  0x00007f635cb54be5 in pthread_join (threadid=140064670824208, thread_return=0x0) at pthread_join.c:89
#1  0x000000000041d6dd in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3304
Current language:  auto
The current source language is "auto; currently c".
-------------- next part --------------
Thread 7 (Thread 0x7f6358f33710 (LWP 24325)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00007f635cb560e9 in _L_lock_953 () from /lib/libpthread.so.0
#2  0x00007f635cb55f0b in __pthread_mutex_lock (mutex=0x7f6350015910) at pthread_mutex_lock.c:61
#3  0x000000000043d44d in qemuDriverLock (conn=0x7f635009b8f0) at qemu/qemu_driver.c:144
#4  qemudClose (conn=0x7f635009b8f0) at qemu/qemu_driver.c:4500
#5  0x00007f636054a97b in virReleaseConnect (conn=0x7f635009b8f0) at datatypes.c:269
#6  0x00007f636054b028 in virUnrefConnect (conn=0x7f635009b8f0) at datatypes.c:321
#7  0x00007f63605525c8 in virConnectClose (conn=0x7f635009b8f0) at libvirt.c:1548
#8  0x000000000041941f in qemudFreeClient (opaque=0xc636f0) at libvirtd.c:2284
#9  qemudRunLoop (opaque=0xc636f0) at libvirtd.c:2358
#10 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#11 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f634effd710 (LWP 24476)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ab0) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
Current language:  auto
The current source language is "auto; currently asm".

Thread 5 (Thread 0x7f634e7fc710 (LWP 24478)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ac8) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f634dffb710 (LWP 24489)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74ae0) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f634d7fa710 (LWP 24494)):
#0  0x00007f635cb5b0bd in read () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f636051683e in read (fd=54, buf=0x13b5db0, count=65536) at /usr/include/bits/unistd.h:45
#2  saferead (fd=54, buf=0x13b5db0, count=65536) at util/util.c:95
#3  0x0000000000447e03 in doTunnelSendAll (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, 
    dom_xml=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, 
    resource=0, dom=<value optimized out>) at qemu/qemu_driver.c:11389
#4  doTunnelMigrate (driver=0x7f6350015910, dconn=0x1202cf0, vm=0x7f63500197b0, dom_xml=<value optimized out>, 
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0, dom=<value optimized out>)
    at qemu/qemu_driver.c:11604
#5  0x0000000000448631 in doPeer2PeerMigrate (dom=0xc63970, cookie=<value optimized out>, 
    cookielen=<value optimized out>, uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, 
    resource=0) at qemu/qemu_driver.c:11746
#6  qemudDomainMigratePerform (dom=0xc63970, cookie=<value optimized out>, cookielen=<value optimized out>, 
---Type <return> to continue, or q <return> to quit---
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, resource=0)
    at qemu/qemu_driver.c:11816
#7  0x00007f6360555b16 in virDomainMigratePerform (domain=0xc63970, cookie=0x0, cookielen=0, 
    uri=0xc65980 "qemu+tcp://10.32.174.165/system", flags=23, dname=0x0, bandwidth=0) at libvirt.c:3931
#8  0x0000000000425782 in remoteDispatchDomainMigratePerform (server=<value optimized out>, 
    client=<value optimized out>, conn=0x7f63501472a0, hdr=<value optimized out>, rerr=0x7f634d7f9be0, 
    args=0x7f634d7f9d20, ret=0x7f634d7f9cc0) at remote.c:1852
#9  0x000000000042b417 in remoteDispatchClientCall (server=0xc636f0, client=0xc66fc0, msg=0xf6d650)
    at dispatch.c:530
#10 remoteDispatchClientRequest (server=0xc636f0, client=0xc66fc0, msg=0xf6d650) at dispatch.c:408
#11 0x000000000041aaa8 in qemudWorker (data=0xc74af8) at libvirtd.c:1586
#12 0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#13 0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f635672e710 (LWP 24853)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f6360515886 in virCondWait (c=<value optimized out>, m=<value optimized out>)
    at util/threads-pthread.c:105
#2  0x000000000041aa15 in qemudWorker (data=0xc74a68) at libvirtd.c:1565
#3  0x00007f635cb538ba in start_thread (arg=<value optimized out>) at pthread_create.c:300
#4  0x00007f635c6b702d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f6360ce07e0 (LWP 24324)):
#0  0x00007f635cb54be5 in pthread_join (threadid=140064670824208, thread_return=0x0) at pthread_join.c:89
#1  0x000000000041d6dd in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3304
Current language:  auto
The current source language is "auto; currently c".
-------------- next part --------------
Thread 1 (Thread 0x7f849b1687e0 (LWP 24979)):
#0  0x00007f84975490d8 in *__GI___poll (fds=0x7fff70e53d90, nfds=2, timeout=<value optimized out>)
    at ../sysdeps/unix/sysv/linux/poll.c:83
#1  0x00007f849abf68e2 in remoteIOEventLoop (conn=0x171a1f0, priv=0x7f849afb2010, flags=3, 
    thiscall=0x7f84949d2010) at remote/remote_driver.c:10129
#2  remoteIO (conn=0x171a1f0, priv=0x7f849afb2010, flags=3, thiscall=0x7f84949d2010)
    at remote/remote_driver.c:10370
#3  0x00007f849abf731c in call (conn=0x171a1f0, priv=0x7f849afb2010, flags=3, proc_nr=<value optimized out>, 
    args_filter=0x7f849757d240 <xdr_void>, args=<value optimized out>, 
    ret_filter=0x7f849ac0c000 <xdr_remote_auth_list_ret>, ret=0x7fff70e54400 "")
    at remote/remote_driver.c:10455
#4  0x00007f849ac070ff in remoteAuthenticate (conn=<value optimized out>, priv=0x7f849afb2010, 
    auth=0x7f849af53340, flags=0) at remote/remote_driver.c:6853
#5  doRemoteOpen (conn=<value optimized out>, priv=0x7f849afb2010, auth=0x7f849af53340, flags=0)
    at remote/remote_driver.c:861
#6  0x00007f849ac08ee0 in remoteOpen (conn=0x171a1f0, auth=0x7f849af53340, flags=<value optimized out>)
    at remote/remote_driver.c:1103
#7  0x00007f849abd6c19 in do_open (name=0x0, auth=0x7f849af53340, flags=0) at libvirt.c:1259
#8  0x00007f849abd8518 in virConnectOpenAuth (name=0x0, auth=0x7f849af53340, flags=0) at libvirt.c:1514
#9  0x000000000041ee17 in vshInit (argc=<value optimized out>, argv=0x7fff70e54af8) at virsh.c:11167
#10 main (argc=<value optimized out>, argv=0x7fff70e54af8) at virsh.c:11819

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20101220/fcf592a4/attachment.sig>

More information about the libvirt-users mailing list