[libvirt-users] libvirt hang on domain attemp to create domain

Александр sss at sss.chaoslab.ru
Thu Jun 20 12:59:56 UTC 2013 

В письме от Четверг, 20-июн-2013 13:25:06 пользователь Michal Privoznik 
написал:
> On 20.06.2013 12:56, Alexandr wrote:
> > On 20.06.2013 10:44, Michal Privoznik wrote:
> >> On 20.06.2013 09:40, Александр wrote:
> >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
> >>>> On 19.06.2013 21:24, Александр wrote:
> >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik
> >>> 
> >>> написал:
> >>>>>> On 18.06.2013 22:21, Александр wrote:
> >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal
> >>>>>>> Privoznik
> >>>>>>> 
> >>>>>>> написал:
> >>>>>>>> On 18.06.2013 07:14, Александр wrote:
> >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML.
> >>>>>>>>> and my
> >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with
> >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and
> >>>>>>>>> restarting
> >>>>>>>>> it solveing hang (i mean after restart i able to connect with
> >>>>>>>>> virt-manager), i using following domain xml
> >>>>>>>>> 
> >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
> >>>>>>>>> <domain type="kvm">
> >>>>>>>>> 
> >>>>>>>>>    <tittle>Automatically created domain with default
> >>>>>>>>>    configuration</tittle>
> >>>>>>>>>    <name>vm1</name>
> >>>>>>>>>    <description>no description</description>
> >>>>>>>>>    <os>
> >>>>>>>>>    
> >>>>>>>>>         <type>hvm</type>
> >>>>>>>>>      
> >>>>>>>>>      <boot dev="cdrom"/>
> >>>>>>>>>      <boot dev="hd"/>
> >>>>>>>>>      
> >>>>>>>>>       </os>
> >>>>>>>>>    
> >>>>>>>>>    <vcpu>1</vcpu>
> >>>>>>>>>    <memory unit="MB">512</memory>
> >>>>>>>>>    <cpu model="host-model">
> >>>>>>>>>    
> >>>>>>>>>         <topology sockets="1" cores="1" threads="1"/>
> >>>>>>>>>       
> >>>>>>>>>       </cpu>
> >>>>>>>>>    
> >>>>>>>>>    <clock offset="localtime"/>
> >>>>>>>>>    <devices>
> >>>>>>>>>    
> >>>>>>>>>         <graphics type="vnc" autoport="yes" password="admin"/>
> >>>>>>>>>      
> >>>>>>>>>      <channel type="unix">
> >>>>>>>>>      
> >>>>>>>>>             <source mode="bind" path="/var/run/vm1.sock"/>
> >>>>>>>>>        
> >>>>>>>>>        <target type="virtio" name="org.guest_agent.0"/>
> >>>>>>>>>        
> >>>>>>>>>           </channel>
> >>>>>>>>>       
> >>>>>>>>>       </devices>
> >>>>>>>>> 
> >>>>>>>>> </domain>
> >>>>>>>>> 
> >>>>>>>>> i have this calls on stack
> >>>>>>>>> 
> >>>>>>>>> 0  0x00007ffff48c1750  __poll_nocancel
> >>>>>>>>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> >>>>>>>> 
> >>>>>>>> This is just probably a client waiting for the corresponding
> >>>>>>>> libvirtd
> >>>>>>>> to
> >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs
> >>>>>>>> so we
> >>>>>>>> can see what's going on?
> >>>>>>>> 
> >>>>>>>> http://wiki.libvirt.org/page/DebugLogs
> >>>>>>>> 
> >>>>>>>> BTW, what's the libvirt version you are running?
> >>>>>>>> 
> >>>>>>>> Michal
> >>>>>>>> 
> >>>>>>>>> 2  0x00007ffff649ed96  virNetClientSendInternal
> >>>>>>>>> 3  0x00007ffff64a0443  virNetClientSendWithReply
> >>>>>>>>> 4  0x00007ffff64a0daa  virNetClientProgramCall
> >>>>>>>>> 5  0x00007ffff6478566  callFull.isra.2
> >>>>>>>>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
> >>>>>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
> >>>>>>>>> 8  0x0000000000426941  virt_domain::create
> >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
> >>>>>>>>> 9  0x000000000040c1a9  main
> >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> >>>>>>>>> 
> >>>>>>>>> what i am doing wrong ?
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> _______________________________________________
> >>>>>>>>> libvirt-users mailing list
> >>>>>>>>> libvirt-users at redhat.com
> >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
> >>>>>>> 
> >>>>>>> i have libvirt version 1.0.6, debug log in attachment.
> >>>>>> 
> >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is
> >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and
> >>>>>> run 't
> >>>>>> a a bt' and post the result? I wonder if we have a deadlock
> >>>>>> somewhere.
> >>>>>> 
> >>>>>> Michal
> >>>>> 
> >>>>> if i correctly understand you, you need debugger backtrace after
> >>>>> libvirtd
> >>>>> stuck ?, if so, here is bt from gdb
> >>>> 
> >>>> You understand correctly. However, simple 'bt' doesn't show what's
> >>>> going
> >>>> on in other threads - libvirtd is multi threaded.
> >>>> 
> >>>>> (gdb) bt
> >>>>> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
> >>>>> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
> >>>>> /usr/lib64/libvirt.so.0 #2  0x00007f00c203ba4d in
> >>>>> virEventRunDefaultImpl
> >>>>> () from
> >>>>> /usr/lib64/libvirt.so.0
> >>>>> #3  0x00007f00c214a5cd in virNetServerRun () from
> >>>>> /usr/lib64/libvirt.so.0
> >>>>> #4  0x00007f00c2aea08a in main ()
> >>>> 
> >>>> So all we see here is the libvirtd event loop. That's why we need 't
> >>>> a a
> >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over all
> >>>> threads.
> >>>> 
> >>>> Michal
> >>> 
> >>> how can i make it ?
> >> 
> >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll
> >> type 't a a bt'.
> >> 
> >> Michal
> > 
> > here is backtrace from all threads:
> > 
> > (gdb) t a a bt
> > 
> > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
> > #0  0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
> > #1  0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
> > #2  0x00007fdd944b7caa in pthread_mutex_lock () from
> > /lib64/libpthread.so.0
> 
> Bingo! So we've a deadlock here. Somebody has forgotten to unlock a
> domain. So as we go over the domain list to check for name (if it exists
> already or not), we deadlock. The process goes like this:
> 
> 1) if adding a domain the the internal list (I should write rather hash
> table where UUID is the key, and the pointer to the internal structure
> representing domain is the value), we search for UUID. Since this is the
> key, we don't have to lock anything (besides the hash table itself).
> 
> 2) Okay, no matching UUID. But what about domain name? We go through the
> table again, over each element, and:
> 2a) lock the element
> 2b) compare the names
> 2c) unlock the element
> 
> And since a domain withing the table is locked (and all none of the
> threads is using it) we deadlock in step 2a) as there's no one to come
> and unlock the domain. Sigh. Hopefully, I'll have some time to dig into
> the log you've provided and get more precise info.
> 
> > #3  0x00007fdd94773861 in virDomainObjListSearchName () from
> > /usr/lib64/libvirt.so.0
> > #4  0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
> > #5  0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from
> > /usr/lib64/libvirt.so.0
> > #6  0x00007fdd9478cd47 in virDomainObjListAdd () from
> > /usr/lib64/libvirt.so.0
> > #7  0x00007fdd875a311c in qemuDomainDefineXML () from
> > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> > #8  0x00007fdd947ead2f in virDomainDefineXML () from
> > /usr/lib64/libvirt.so.0
> > #9  0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
> > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from
> > /usr/lib64/libvirt.so.0
> > #11 0x00007fdd94844068 in virNetServerHandleJob () from
> > /usr/lib64/libvirt.so.0
> > #12 0x00007fdd94765795 in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > ---Type <return> to continue, or q <return> to quit---
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
> > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > /lib64/libpthread.so.0
> > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > 
> > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
> > #0  0x00007fdd941e376d in poll () from /lib64/libc.so.6
> > #1  0x00007fdd94737fcb in virEventPollRunOnce () from
> > /usr/lib64/libvirt.so.0
> > #2  0x00007fdd94736a4d in virEventRunDefaultImpl () from
> > /usr/lib64/libvirt.so.0
> > #3  0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0
> > #4  0x00007fdd951e508a in main ()
> > 
> > and also from my program if it important:
> > 
> > 0  0x00007ffff48c1750  __poll_nocancel
> > 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> > 2  0x00007ffff649ed96  virNetClientSendInternal
> > 3  0x00007ffff64a0443  virNetClientSendWithReply
> > 4  0x00007ffff64a0daa  virNetClientProgramCall
> > 5  0x00007ffff6478566  callFull.isra.2
> > 6  0x00007ffff6487fa2  remoteDomainDefineXML
> > 7  0x00007ffff6449d2f  virDomainDefineXML
> > 8  0x0000000000426941  virt_domain::create
> > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
> > 9  0x000000000040c1a9  main
> > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> 
> Michal

thx for quick reply, i will wait.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20130620/aa3ab02f/attachment.sig>

More information about the libvirt-users mailing list