[libvirt] [PATCH] [RFC] nwfilter: resolve deadlock between VM operations and filter update

Stefan Berger stefanb at linux.vnet.ibm.com
Thu Oct 7 13:58:28 UTC 2010


  On 10/07/2010 09:06 AM, Soren Hansen wrote:
> I had trouble applying the patch (I think maybe Thunderbird may have
> fiddled with the formatting :( ), but after doing it manually, it works
> excellently. Thanks!
>
Great. I will prepare a V3.

I am also shooting a kill -SIGHUP at libvirt once in a while to see what 
happens (while creating / destroying 2 VMs and modifying their filters). 
Most of the time all goes well, but occasionally things do get stuck. I 
get the following debugging output from libvirt and attaching gdb to 
libvirt I see the following stack traces. Maybe Daniel can interpret 
this... To me it looks like some of the conditions need to be 'tickled'...


09:47:25.000: error : qemuAutostartDomain:822 : Failed to start job on 
VM 'dummy-vm1': Timed out during operation: cannot acquire state change lock

(gdb) thr ap all bt

Thread 9 (Thread 0x7f49bf592710 (LWP 17464)):
#0  0x000000327680b729 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x0000000000435312 in virCondWaitUntil (c=<value optimized out>,
     m=<value optimized out>, whenms=<value optimized out>)
     at util/threads-pthread.c:115
#2  0x000000000043d0ab in qemuDomainObjBeginJobWithDriver 
(driver=0x1f9c010,
     obj=0x7f49a00011b0) at qemu/qemu_driver.c:409
#3  0x0000000000458abf in qemuAutostartDomain (payload=<value optimized 
out>,
     name=<value optimized out>, opaque=0x7f49bf591320)
     at qemu/qemu_driver.c:818
#4  0x00007f49c040ab6a in virHashForEach (table=0x1f9be20,
     iter=0x458a90 <qemuAutostartDomain>, data=0x7f49bf591320)
     at util/hash.c:495
#5  0x000000000043cdac in qemudAutostartConfigs (driver=0x1f9c010)
     at qemu/qemu_driver.c:855
#6  0x000000000043ce2a in qemudReload () at qemu/qemu_driver.c:2003
#7  0x00007f49c0450a3e in virStateReload () at libvirt.c:1017
#8  0x00000000004189e1 in qemudDispatchSignalEvent (
     watch=<value optimized out>, fd=<value optimized out>,
     events=<value optimized out>, opaque=0x1f6f830) at libvirtd.c:388
---Type <return> to continue, or q <return> to quit---
#9  0x00000000004186a9 in virEventDispatchHandles () at event.c:479
#10 virEventRunOnce () at event.c:608
#11 0x000000000041a346 in qemudOneLoop () at libvirtd.c:2217
#12 0x000000000041a613 in qemudRunLoop (opaque=0x1f6f830) at libvirtd.c:2326
#13 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#14 0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f49beb91710 (LWP 17465)):
#0  0x000000327680b3bc in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x0000000000435336 in virCondWait (c=<value optimized out>,
     m=<value optimized out>) at util/threads-pthread.c:100
#2  0x000000000041b2e5 in qemudWorker (data=0x7f49b80008c0) at 
libvirtd.c:1549
#3  0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#4  0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f49be190710 (LWP 17466)):
#0  0x000000327680b3bc in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x0000000000435336 in virCondWait (c=<value optimized out>,
     m=<value optimized out>) at util/threads-pthread.c:100
#2  0x00000000004716f6 in qemuMonitorSend (mon=0x7f49a00040b0,
---Type <return> to continue, or q <return> to quit---
     msg=<value optimized out>) at qemu/qemu_monitor.c:726
#3  0x0000000000473673 in qemuMonitorCommandWithHandler 
(mon=0x7f49a00040b0,
     cmd=0x4c316f "info cpus", passwordHandler=0, passwordOpaque=0x0,
     scm_fd=-1, reply=0x7f49be18e400) at qemu/qemu_monitor_text.c:233
#4  0x000000000047845e in qemuMonitorTextGetCPUInfo (
     mon=<value optimized out>, pids=0x7f49be18fa18)
     at qemu/qemu_monitor_text.c:401
#5  0x0000000000455db2 in qemuDetectVcpuPIDs (conn=0x7f49ac036100,
     driver=0x1f9c010, vm=<value optimized out>, migrateFrom=0x0,
     start_paused=false, stdin_fd=-1, stdin_path=0x0) at 
qemu/qemu_driver.c:2413
#6  qemudStartVMDaemon (conn=0x7f49ac036100, driver=0x1f9c010,
     vm=<value optimized out>, migrateFrom=0x0, start_paused=false,
     stdin_fd=-1, stdin_path=0x0) at qemu/qemu_driver.c:3925
#7  0x0000000000457f6b in qemudDomainCreate (conn=0x7f49ac036100,
     xml=<value optimized out>, flags=0) at qemu/qemu_driver.c:4559
#8  0x00007f49c04564ab in virDomainCreateXML (conn=0x7f49ac036100,
     xmlDesc=0x7f49a02b8bf0 " <domain type='kvm'>\n 
<name>dummy-vm2</name>\n <memory>32768</memory>\n 
<currentMemory>32768</currentMemory>\n <vcpu>1</vcpu>\n <os>\n 
<type>hvm</type>\n <boot dev='hd'/>\n </os>"..., flags=0) at libvirt.c:1984
#9  0x0000000000427da8 in remoteDispatchDomainCreateXml (
     server=<value optimized out>, client=<value optimized out>,
---Type <return> to continue, or q <return> to quit---
     conn=0x7f49ac036100, hdr=<value optimized out>, rerr=0x7f49be18fc70,
     args=<value optimized out>, ret=0x7f49be18fbc0) at remote.c:1271
#10 0x000000000042a657 in remoteDispatchClientCall (server=0x1f6f830,
     client=0x7f49b83f03c0, msg=0x7f49b8453800) at dispatch.c:529
#11 remoteDispatchClientRequest (server=0x1f6f830, client=0x7f49b83f03c0,
     msg=0x7f49b8453800) at dispatch.c:407
#12 0x000000000041b378 in qemudWorker (data=0x7f49b80008d8) at 
libvirtd.c:1570
#13 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#14 0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f49bd78f710 (LWP 17467)):
#0  0x000000327680b3bc in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x0000000000435336 in virCondWait (c=<value optimized out>,
     m=<value optimized out>) at util/threads-pthread.c:100
#2  0x000000000041b2e5 in qemudWorker (data=0x7f49b80008f0) at 
libvirtd.c:1549
#3  0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#4  0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f49bcd8e710 (LWP 17468)):
#0  0x000000327680b3bc in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x0000000000435336 in virCondWait (c=<value optimized out>,
     m=<value optimized out>) at util/threads-pthread.c:100
#2  0x00000000004716f6 in qemuMonitorSend (mon=0x7f49ac099740,
     msg=<value optimized out>) at qemu/qemu_monitor.c:726
#3  0x0000000000473673 in qemuMonitorCommandWithHandler 
(mon=0x7f49ac099740,
     cmd=0x7f49ac381620 "balloon 32", passwordHandler=0, 
passwordOpaque=0x0,
     scm_fd=-1, reply=0x7f49bcd8c420) at qemu/qemu_monitor_text.c:233
#4  0x0000000000475f45 in qemuMonitorTextSetBalloon (mon=0x7f49ac099740,
     newmem=<value optimized out>) at qemu/qemu_monitor_text.c:791
#5  0x0000000000455743 in qemudStartVMDaemon (conn=0x7f49a811f9c0,
     driver=0x1f9c010, vm=<value optimized out>, migrateFrom=0x0,
     start_paused=false, stdin_fd=-1, stdin_path=0x0) at 
qemu/qemu_driver.c:3942
#6  0x0000000000457f6b in qemudDomainCreate (conn=0x7f49a811f9c0,
     xml=<value optimized out>, flags=0) at qemu/qemu_driver.c:4559
#7  0x00007f49c04564ab in virDomainCreateXML (conn=0x7f49a811f9c0,
     xmlDesc=0x7f49ac2f9f20 " <domain type='kvm'>\n 
<name>dummy-vm1</name>\n <memory>32768</memory>\n 
<currentMemory>32768</currentMemory>\n <vcpu>1</vcpu>\n <os>\n 
<type>hvm</type>\n <boot dev='hd'/>\n </os>"..., flags=0) at libvirt.c:1984
#8  0x0000000000427da8 in remoteDispatchDomainCreateXml (
     server=<value optimized out>, client=<value optimized out>,
     conn=0x7f49a811f9c0, hdr=<value optimized out>, rerr=0x7f49bcd8dc70,
---Type <return> to continue, or q <return> to quit---
     args=<value optimized out>, ret=0x7f49bcd8dbc0) at remote.c:1271
#9  0x000000000042a657 in remoteDispatchClientCall (server=0x1f6f830,
     client=0x7f49b89b41d0, msg=0x7f49b8306460) at dispatch.c:529
#10 remoteDispatchClientRequest (server=0x1f6f830, client=0x7f49b89b41d0,
     msg=0x7f49b8306460) at dispatch.c:407
#11 0x000000000041b378 in qemudWorker (data=0x7f49b8000908) at 
libvirtd.c:1570
#12 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#13 0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f49b7fff710 (LWP 17469)):
#0  0x000000327680b3bc in pthread_cond_wait@@GLIBC_2.3.2 ()
    from /lib64/libpthread.so.0
#1  0x0000000000435336 in virCondWait (c=<value optimized out>,
     m=<value optimized out>) at util/threads-pthread.c:100
#2  0x000000000041b2e5 in qemudWorker (data=0x7f49b8000920) at 
libvirtd.c:1549
#3  0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#4  0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f486e22b710 (LWP 4599)):
#0  0x00000032760d7de3 in poll () from /lib64/libc.so.6
#1  0x0000003279c07a02 in ?? () from /usr/lib64/libpcap.so.1
#2  0x0000003279c0c65b in pcap_next () from /usr/lib64/libpcap.so.1
---Type <return> to continue, or q <return> to quit---
#3  0x00000000004afce4 in learnIPAddressThread (arg=0x7f49b82de290)
     at nwfilter/nwfilter_learnipaddr.c:496
#4  0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f486d82a710 (LWP 4897)):
#0  0x00000032760d7de3 in poll () from /lib64/libc.so.6
#1  0x0000003279c07a02 in ?? () from /usr/lib64/libpcap.so.1
#2  0x0000003279c0c65b in pcap_next () from /usr/lib64/libpcap.so.1
#3  0x00000000004afce4 in learnIPAddressThread (arg=0x7f49b8802310)
     at nwfilter/nwfilter_learnipaddr.c:496
#4  0x0000003276807761 in start_thread () from /lib64/libpthread.so.0
#5  0x00000032760e14ed in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f49c03c27e0 (LWP 17460)):
#0  0x0000003276807fbd in pthread_join () from /lib64/libpthread.so.0
#1  0x000000000041e434 in main (argc=<value optimized out>,
     argv=<value optimized out>) at libvirtd.c:3272
(gdb)




More information about the libvir-list mailing list