[libvirt-users] libvirt hang on domain attemp to create domain

Александр sss at sss.chaoslab.ru
Mon Jun 24 09:20:00 UTC 2013 

В письме от Четверг, 20-июн-2013 15:59:56 пользователь Александр написал:
> В письме от Четверг, 20-июн-2013 13:25:06 пользователь Michal Privoznik
> 
> написал:
> > On 20.06.2013 12:56, Alexandr wrote:
> > > On 20.06.2013 10:44, Michal Privoznik wrote:
> > >> On 20.06.2013 09:40, Александр wrote:
> > >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
> > >>>> On 19.06.2013 21:24, Александр wrote:
> > >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal
> > >>>>> Privoznik
> > >>> 
> > >>> написал:
> > >>>>>> On 18.06.2013 22:21, Александр wrote:
> > >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal
> > >>>>>>> Privoznik
> > >>>>>>> 
> > >>>>>>> написал:
> > >>>>>>>> On 18.06.2013 07:14, Александр wrote:
> > >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML.
> > >>>>>>>>> and my
> > >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon
> > >>>>>>>>> with
> > >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and
> > >>>>>>>>> restarting
> > >>>>>>>>> it solveing hang (i mean after restart i able to connect with
> > >>>>>>>>> virt-manager), i using following domain xml
> > >>>>>>>>> 
> > >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
> > >>>>>>>>> <domain type="kvm">
> > >>>>>>>>> 
> > >>>>>>>>>    <tittle>Automatically created domain with default
> > >>>>>>>>>    configuration</tittle>
> > >>>>>>>>>    <name>vm1</name>
> > >>>>>>>>>    <description>no description</description>
> > >>>>>>>>>    <os>
> > >>>>>>>>>    
> > >>>>>>>>>         <type>hvm</type>
> > >>>>>>>>>      
> > >>>>>>>>>      <boot dev="cdrom"/>
> > >>>>>>>>>      <boot dev="hd"/>
> > >>>>>>>>>      
> > >>>>>>>>>       </os>
> > >>>>>>>>>    
> > >>>>>>>>>    <vcpu>1</vcpu>
> > >>>>>>>>>    <memory unit="MB">512</memory>
> > >>>>>>>>>    <cpu model="host-model">
> > >>>>>>>>>    
> > >>>>>>>>>         <topology sockets="1" cores="1" threads="1"/>
> > >>>>>>>>>       
> > >>>>>>>>>       </cpu>
> > >>>>>>>>>    
> > >>>>>>>>>    <clock offset="localtime"/>
> > >>>>>>>>>    <devices>
> > >>>>>>>>>    
> > >>>>>>>>>         <graphics type="vnc" autoport="yes" password="admin"/>
> > >>>>>>>>>      
> > >>>>>>>>>      <channel type="unix">
> > >>>>>>>>>      
> > >>>>>>>>>             <source mode="bind" path="/var/run/vm1.sock"/>
> > >>>>>>>>>        
> > >>>>>>>>>        <target type="virtio" name="org.guest_agent.0"/>
> > >>>>>>>>>        
> > >>>>>>>>>           </channel>
> > >>>>>>>>>       
> > >>>>>>>>>       </devices>
> > >>>>>>>>> 
> > >>>>>>>>> </domain>
> > >>>>>>>>> 
> > >>>>>>>>> i have this calls on stack
> > >>>>>>>>> 
> > >>>>>>>>> 0  0x00007ffff48c1750  __poll_nocancel
> > >>>>>>>>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> > >>>>>>>> 
> > >>>>>>>> This is just probably a client waiting for the corresponding
> > >>>>>>>> libvirtd
> > >>>>>>>> to
> > >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs
> > >>>>>>>> so we
> > >>>>>>>> can see what's going on?
> > >>>>>>>> 
> > >>>>>>>> http://wiki.libvirt.org/page/DebugLogs
> > >>>>>>>> 
> > >>>>>>>> BTW, what's the libvirt version you are running?
> > >>>>>>>> 
> > >>>>>>>> Michal
> > >>>>>>>> 
> > >>>>>>>>> 2  0x00007ffff649ed96  virNetClientSendInternal
> > >>>>>>>>> 3  0x00007ffff64a0443  virNetClientSendWithReply
> > >>>>>>>>> 4  0x00007ffff64a0daa  virNetClientProgramCall
> > >>>>>>>>> 5  0x00007ffff6478566  callFull.isra.2
> > >>>>>>>>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
> > >>>>>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
> > >>>>>>>>> 8  0x0000000000426941  virt_domain::create
> > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 
> > >>>>>>>>> 295
> > >>>>>>>>> 9  0x000000000040c1a9  main
> > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> > >>>>>>>>> 
> > >>>>>>>>> what i am doing wrong ?
> > >>>>>>>>> 
> > >>>>>>>>> 
> > >>>>>>>>> 
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> libvirt-users mailing list
> > >>>>>>>>> libvirt-users at redhat.com
> > >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
> > >>>>>>> 
> > >>>>>>> i have libvirt version 1.0.6, debug log in attachment.
> > >>>>>> 
> > >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon
> > >>>>>> is
> > >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and
> > >>>>>> run 't
> > >>>>>> a a bt' and post the result? I wonder if we have a deadlock
> > >>>>>> somewhere.
> > >>>>>> 
> > >>>>>> Michal
> > >>>>> 
> > >>>>> if i correctly understand you, you need debugger backtrace after
> > >>>>> libvirtd
> > >>>>> stuck ?, if so, here is bt from gdb
> > >>>> 
> > >>>> You understand correctly. However, simple 'bt' doesn't show what's
> > >>>> going
> > >>>> on in other threads - libvirtd is multi threaded.
> > >>>> 
> > >>>>> (gdb) bt
> > >>>>> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
> > >>>>> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
> > >>>>> /usr/lib64/libvirt.so.0 #2  0x00007f00c203ba4d in
> > >>>>> virEventRunDefaultImpl
> > >>>>> () from
> > >>>>> /usr/lib64/libvirt.so.0
> > >>>>> #3  0x00007f00c214a5cd in virNetServerRun () from
> > >>>>> /usr/lib64/libvirt.so.0
> > >>>>> #4  0x00007f00c2aea08a in main ()
> > >>>> 
> > >>>> So all we see here is the libvirtd event loop. That's why we need 't
> > >>>> a a
> > >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over
> > >>>> all
> > >>>> threads.
> > >>>> 
> > >>>> Michal
> > >>> 
> > >>> how can i make it ?
> > >> 
> > >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll
> > >> type 't a a bt'.
> > >> 
> > >> Michal
> > > 
> > > here is backtrace from all threads:
> > > 
> > > (gdb) t a a bt
> > > 
> > > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
> > > #0  0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
> > > #1  0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
> > > #2  0x00007fdd944b7caa in pthread_mutex_lock () from
> > > /lib64/libpthread.so.0
> > 
> > Bingo! So we've a deadlock here. Somebody has forgotten to unlock a
> > domain. So as we go over the domain list to check for name (if it exists
> > already or not), we deadlock. The process goes like this:
> > 
> > 1) if adding a domain the the internal list (I should write rather hash
> > table where UUID is the key, and the pointer to the internal structure
> > representing domain is the value), we search for UUID. Since this is the
> > key, we don't have to lock anything (besides the hash table itself).
> > 
> > 2) Okay, no matching UUID. But what about domain name? We go through the
> > table again, over each element, and:
> > 2a) lock the element
> > 2b) compare the names
> > 2c) unlock the element
> > 
> > And since a domain withing the table is locked (and all none of the
> > threads is using it) we deadlock in step 2a) as there's no one to come
> > and unlock the domain. Sigh. Hopefully, I'll have some time to dig into
> > the log you've provided and get more precise info.
> > 
> > > #3  0x00007fdd94773861 in virDomainObjListSearchName () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
> > > #5  0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from
> > > /usr/lib64/libvirt.so.0
> > > #6  0x00007fdd9478cd47 in virDomainObjListAdd () from
> > > /usr/lib64/libvirt.so.0
> > > #7  0x00007fdd875a311c in qemuDomainDefineXML () from
> > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> > > #8  0x00007fdd947ead2f in virDomainDefineXML () from
> > > /usr/lib64/libvirt.so.0
> > > #9  0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
> > > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from
> > > /usr/lib64/libvirt.so.0
> > > #11 0x00007fdd94844068 in virNetServerHandleJob () from
> > > /usr/lib64/libvirt.so.0
> > > #12 0x00007fdd94765795 in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #13 0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476582b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > ---Type <return> to continue, or q <return> to quit---
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
> > > #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> > > /lib64/libpthread.so.0
> > > #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd9476584b in virThreadPoolWorker () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd94764dfe in virThreadHelper () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> > > #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> > > 
> > > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
> > > #0  0x00007fdd941e376d in poll () from /lib64/libc.so.6
> > > #1  0x00007fdd94737fcb in virEventPollRunOnce () from
> > > /usr/lib64/libvirt.so.0
> > > #2  0x00007fdd94736a4d in virEventRunDefaultImpl () from
> > > /usr/lib64/libvirt.so.0
> > > #3  0x00007fdd948455cd in virNetServerRun () from
> > > /usr/lib64/libvirt.so.0
> > > #4  0x00007fdd951e508a in main ()
> > > 
> > > and also from my program if it important:
> > > 
> > > 0  0x00007ffff48c1750  __poll_nocancel
> > > 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> > > 2  0x00007ffff649ed96  virNetClientSendInternal
> > > 3  0x00007ffff64a0443  virNetClientSendWithReply
> > > 4  0x00007ffff64a0daa  virNetClientProgramCall
> > > 5  0x00007ffff6478566  callFull.isra.2
> > > 6  0x00007ffff6487fa2  remoteDomainDefineXML
> > > 7  0x00007ffff6449d2f  virDomainDefineXML
> > > 8  0x0000000000426941  virt_domain::create
> > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
> > > 9  0x000000000040c1a9  main
> > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> > 
> > Michal
> 
> thx for quick reply, i will wait.

i have tried to workaround this, added uuid to xml, but nothing changed, 
currently my xml looks like 

<?xml version="1.0" encoding="UTF-8"?>
<domain type="kvm">
  <tittle>Automatically created domain with default configuration</tittle>
  <uuid>2961e3ca-6318-4af7-85e5-72affde5dc6e</uuid>
  <name>vm1</name>
  <description>no description</description>
  <os>
    <type>hvm</type>
    <boot dev="cdrom"/>
    <boot dev="hd"/>
  </os>
  <vcpu>1</vcpu>
  <memory unit="MB">512</memory>
  <cpu model="host-model">
    <topology sockets="1" cores="1" threads="1"/>
  </cpu>
  <clock offset="localtime"/>
  <devices>
    <graphics type="vnc" autoport="yes" password="admin"/>
    <channel type="unix">
      <source mode="bind" path="/var/run/vm1.sock"/>
      <target type="virtio" name="org.guest_agent.0"/>
    </channel>
  </devices>
</domain>

but virt-manager can create domain, how can i find difference in configs ?, 
how can i obtain config from virt manager (without deep research of code)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20130624/c0720b34/attachment.sig>

More information about the libvirt-users mailing list