[libvirt-users] libvirt hang on domain attemp to create domain

Alexandr sss at sss.chaoslab.ru
Thu Jun 20 10:56:15 UTC 2013 

On 20.06.2013 10:44, Michal Privoznik wrote:
> On 20.06.2013 09:40, Александр wrote:
>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
>>> On 19.06.2013 21:24, Александр wrote:
>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik
>> написал:
>>>>> On 18.06.2013 22:21, Александр wrote:
>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal Privoznik
>>>>>>
>>>>>> написал:
>>>>>>> On 18.06.2013 07:14, Александр wrote:
>>>>>>>> good day, i am trying to create domain via virDomainDefineXML. and my
>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with
>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and
>>>>>>>> restarting
>>>>>>>> it solveing hang (i mean after restart i able to connect with
>>>>>>>> virt-manager), i using following domain xml
>>>>>>>>
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <domain type="kvm">
>>>>>>>>
>>>>>>>>    <tittle>Automatically created domain with default
>>>>>>>>    configuration</tittle>
>>>>>>>>    <name>vm1</name>
>>>>>>>>    <description>no description</description>
>>>>>>>>    <os>
>>>>>>>>    
>>>>>>>>      <type>hvm</type>
>>>>>>>>      <boot dev="cdrom"/>
>>>>>>>>      <boot dev="hd"/>
>>>>>>>>    
>>>>>>>>    </os>
>>>>>>>>    <vcpu>1</vcpu>
>>>>>>>>    <memory unit="MB">512</memory>
>>>>>>>>    <cpu model="host-model">
>>>>>>>>    
>>>>>>>>      <topology sockets="1" cores="1" threads="1"/>
>>>>>>>>    
>>>>>>>>    </cpu>
>>>>>>>>    <clock offset="localtime"/>
>>>>>>>>    <devices>
>>>>>>>>    
>>>>>>>>      <graphics type="vnc" autoport="yes" password="admin"/>
>>>>>>>>      <channel type="unix">
>>>>>>>>      
>>>>>>>>        <source mode="bind" path="/var/run/vm1.sock"/>
>>>>>>>>        <target type="virtio" name="org.guest_agent.0"/>
>>>>>>>>      
>>>>>>>>      </channel>
>>>>>>>>    
>>>>>>>>    </devices>
>>>>>>>>
>>>>>>>> </domain>
>>>>>>>>
>>>>>>>> i have this calls on stack
>>>>>>>>
>>>>>>>> 0  0x00007ffff48c1750  __poll_nocancel
>>>>>>>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
>>>>>>> This is just probably a client waiting for the corresponding libvirtd
>>>>>>> to
>>>>>>> provide a reply message. Are you able to get libvirtd debug logs so we
>>>>>>> can see what's going on?
>>>>>>>
>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>
>>>>>>> BTW, what's the libvirt version you are running?
>>>>>>>
>>>>>>> Michal
>>>>>>>
>>>>>>>> 2  0x00007ffff649ed96  virNetClientSendInternal
>>>>>>>> 3  0x00007ffff64a0443  virNetClientSendWithReply
>>>>>>>> 4  0x00007ffff64a0daa  virNetClientProgramCall
>>>>>>>> 5  0x00007ffff6478566  callFull.isra.2
>>>>>>>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
>>>>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
>>>>>>>> 8  0x0000000000426941  virt_domain::create
>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
>>>>>>>> 9  0x000000000040c1a9  main
>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
>>>>>>>>
>>>>>>>> what i am doing wrong ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> libvirt-users mailing list
>>>>>>>> libvirt-users at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>> i have libvirt version 1.0.6, debug log in attachment.
>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is
>>>>> stuck. Can you attach to the libvirtd while it's being stuck and run 't
>>>>> a a bt' and post the result? I wonder if we have a deadlock somewhere.
>>>>>
>>>>> Michal
>>>> if i correctly understand you, you need debugger backtrace after libvirtd
>>>> stuck ?, if so, here is bt from gdb
>>> You understand correctly. However, simple 'bt' doesn't show what's going
>>> on in other threads - libvirtd is multi threaded.
>>>
>>>> (gdb) bt
>>>> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
>>>> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
>>>> /usr/lib64/libvirt.so.0 #2  0x00007f00c203ba4d in virEventRunDefaultImpl
>>>> () from
>>>> /usr/lib64/libvirt.so.0
>>>> #3  0x00007f00c214a5cd in virNetServerRun () from /usr/lib64/libvirt.so.0
>>>> #4  0x00007f00c2aea08a in main ()
>>> So all we see here is the libvirtd event loop. That's why we need 't a a
>>> bt' which is 'thread apply all backtrace' which will run 'bt' over all
>>> threads.
>>>
>>> Michal
>> how can i make it ?
>>
> The very same way as you did 'bt'. Just instead of typing 'bt' you'll
> type 't a a bt'.
>
> Michal
here is backtrace from all threads:

(gdb) t a a bt

Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
#0  0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
#2  0x00007fdd944b7caa in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fdd94773861 in virDomainObjListSearchName () from 
/usr/lib64/libvirt.so.0
#4  0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
#5  0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from 
/usr/lib64/libvirt.so.0
#6  0x00007fdd9478cd47 in virDomainObjListAdd () from 
/usr/lib64/libvirt.so.0
#7  0x00007fdd875a311c in qemuDomainDefineXML () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007fdd947ead2f in virDomainDefineXML () from /usr/lib64/libvirt.so.0
#9  0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
#10 0x00007fdd94848d27 in virNetServerProgramDispatch () from 
/usr/lib64/libvirt.so.0
#11 0x00007fdd94844068 in virNetServerHandleJob () from 
/usr/lib64/libvirt.so.0
#12 0x00007fdd94765795 in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
#0  0x00007fdd941e376d in poll () from /lib64/libc.so.6
#1  0x00007fdd94737fcb in virEventPollRunOnce () from 
/usr/lib64/libvirt.so.0
#2  0x00007fdd94736a4d in virEventRunDefaultImpl () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0
#4  0x00007fdd951e508a in main ()

and also from my program if it important:

0  0x00007ffff48c1750  __poll_nocancel
1  0x00007ffff649e5ca  virNetClientIOEventLoop
2  0x00007ffff649ed96  virNetClientSendInternal
3  0x00007ffff64a0443  virNetClientSendWithReply
4  0x00007ffff64a0daa  virNetClientProgramCall
5  0x00007ffff6478566  callFull.isra.2
6  0x00007ffff6487fa2  remoteDomainDefineXML
7  0x00007ffff6449d2f  virDomainDefineXML
8  0x0000000000426941  virt_domain::create 
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
9  0x000000000040c1a9  main 
/home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40

More information about the libvirt-users mailing list