[libvirt-users] libvirt hang on domain attemp to create domain
Alexandr
sss at sss.chaoslab.ru
Thu Jun 20 10:56:15 UTC 2013
On 20.06.2013 10:44, Michal Privoznik wrote:
> On 20.06.2013 09:40, Александр wrote:
>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
>>> On 19.06.2013 21:24, Александр wrote:
>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik
>> написал:
>>>>> On 18.06.2013 22:21, Александр wrote:
>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal Privoznik
>>>>>>
>>>>>> написал:
>>>>>>> On 18.06.2013 07:14, Александр wrote:
>>>>>>>> good day, i am trying to create domain via virDomainDefineXML. and my
>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with
>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and
>>>>>>>> restarting
>>>>>>>> it solveing hang (i mean after restart i able to connect with
>>>>>>>> virt-manager), i using following domain xml
>>>>>>>>
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <domain type="kvm">
>>>>>>>>
>>>>>>>> <tittle>Automatically created domain with default
>>>>>>>> configuration</tittle>
>>>>>>>> <name>vm1</name>
>>>>>>>> <description>no description</description>
>>>>>>>> <os>
>>>>>>>>
>>>>>>>> <type>hvm</type>
>>>>>>>> <boot dev="cdrom"/>
>>>>>>>> <boot dev="hd"/>
>>>>>>>>
>>>>>>>> </os>
>>>>>>>> <vcpu>1</vcpu>
>>>>>>>> <memory unit="MB">512</memory>
>>>>>>>> <cpu model="host-model">
>>>>>>>>
>>>>>>>> <topology sockets="1" cores="1" threads="1"/>
>>>>>>>>
>>>>>>>> </cpu>
>>>>>>>> <clock offset="localtime"/>
>>>>>>>> <devices>
>>>>>>>>
>>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/>
>>>>>>>> <channel type="unix">
>>>>>>>>
>>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/>
>>>>>>>> <target type="virtio" name="org.guest_agent.0"/>
>>>>>>>>
>>>>>>>> </channel>
>>>>>>>>
>>>>>>>> </devices>
>>>>>>>>
>>>>>>>> </domain>
>>>>>>>>
>>>>>>>> i have this calls on stack
>>>>>>>>
>>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel
>>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop
>>>>>>> This is just probably a client waiting for the corresponding libvirtd
>>>>>>> to
>>>>>>> provide a reply message. Are you able to get libvirtd debug logs so we
>>>>>>> can see what's going on?
>>>>>>>
>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>
>>>>>>> BTW, what's the libvirt version you are running?
>>>>>>>
>>>>>>> Michal
>>>>>>>
>>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal
>>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply
>>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall
>>>>>>>> 5 0x00007ffff6478566 callFull.isra.2
>>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML
>>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML
>>>>>>>> 8 0x0000000000426941 virt_domain::create
>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295
>>>>>>>> 9 0x000000000040c1a9 main
>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40
>>>>>>>>
>>>>>>>> what i am doing wrong ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> libvirt-users mailing list
>>>>>>>> libvirt-users at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>> i have libvirt version 1.0.6, debug log in attachment.
>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is
>>>>> stuck. Can you attach to the libvirtd while it's being stuck and run 't
>>>>> a a bt' and post the result? I wonder if we have a deadlock somewhere.
>>>>>
>>>>> Michal
>>>> if i correctly understand you, you need debugger backtrace after libvirtd
>>>> stuck ?, if so, here is bt from gdb
>>> You understand correctly. However, simple 'bt' doesn't show what's going
>>> on in other threads - libvirtd is multi threaded.
>>>
>>>> (gdb) bt
>>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6
>>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from
>>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in virEventRunDefaultImpl
>>>> () from
>>>> /usr/lib64/libvirt.so.0
>>>> #3 0x00007f00c214a5cd in virNetServerRun () from /usr/lib64/libvirt.so.0
>>>> #4 0x00007f00c2aea08a in main ()
>>> So all we see here is the libvirtd event loop. That's why we need 't a a
>>> bt' which is 'thread apply all backtrace' which will run 'bt' over all
>>> threads.
>>>
>>> Michal
>> how can i make it ?
>>
> The very same way as you did 'bt'. Just instead of typing 'bt' you'll
> type 't a a bt'.
>
> Michal
here is backtrace from all threads:
(gdb) t a a bt
Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476582b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476582b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
#0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
#2 0x00007fdd944b7caa in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fdd94773861 in virDomainObjListSearchName () from
/usr/lib64/libvirt.so.0
#4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
#5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from
/usr/lib64/libvirt.so.0
#6 0x00007fdd9478cd47 in virDomainObjListAdd () from
/usr/lib64/libvirt.so.0
#7 0x00007fdd875a311c in qemuDomainDefineXML () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007fdd947ead2f in virDomainDefineXML () from /usr/lib64/libvirt.so.0
#9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
#10 0x00007fdd94848d27 in virNetServerProgramDispatch () from
/usr/lib64/libvirt.so.0
#11 0x00007fdd94844068 in virNetServerHandleJob () from
/usr/lib64/libvirt.so.0
#12 0x00007fdd94765795 in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476582b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476582b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476584b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476584b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476584b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476584b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
#0 0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2 0x00007fdd9476584b in virThreadPoolWorker () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
#0 0x00007fdd941e376d in poll () from /lib64/libc.so.6
#1 0x00007fdd94737fcb in virEventPollRunOnce () from
/usr/lib64/libvirt.so.0
#2 0x00007fdd94736a4d in virEventRunDefaultImpl () from
/usr/lib64/libvirt.so.0
#3 0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0
#4 0x00007fdd951e508a in main ()
and also from my program if it important:
0 0x00007ffff48c1750 __poll_nocancel
1 0x00007ffff649e5ca virNetClientIOEventLoop
2 0x00007ffff649ed96 virNetClientSendInternal
3 0x00007ffff64a0443 virNetClientSendWithReply
4 0x00007ffff64a0daa virNetClientProgramCall
5 0x00007ffff6478566 callFull.isra.2
6 0x00007ffff6487fa2 remoteDomainDefineXML
7 0x00007ffff6449d2f virDomainDefineXML
8 0x0000000000426941 virt_domain::create
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295
9 0x000000000040c1a9 main
/home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40
More information about the libvirt-users
mailing list