<div dir="ltr">Hi Rich,<div><br></div><div>I spend several days on this issue but still cannot figure out the root cause. Sometimes, guestfs_lauch() gets stuck, sometimes, it is ok.</div><div>I reproduced this issue using guestfish:</div><div><div><font face="monospace, monospace">-bash-4.2# guestfish </font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Welcome to guestfish, the guest filesystem shell for</font></div><div><font face="monospace, monospace">editing virtual machine filesystems and disk images.</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Type: 'help' for help on commands</font></div><div><font face="monospace, monospace"> 'man' to read the manual</font></div><div><font face="monospace, monospace"> 'quit' to quit the shell</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">><fs> add /dev/xvdc</font></div><div><font face="monospace, monospace">><fs> run</font></div><div><font face="monospace, monospace">| 75% [#####################################################################################################################################################################--------------------------------------------------------] 00:15</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">It has been stuck for about 10 minutes. I can see if is in guestfs_launch():</font></div><div><div><font face="monospace, monospace">-bash-4.2# gdb -p `pidof guestfish` -ex "set confirm off" -ex "bt" -ex "q"</font></div><div><div><font face="monospace, monospace">0x00007fe6f6b5a680 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:81</font></div><div><font face="monospace, monospace">81<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)</font></div><div><font face="monospace, monospace">#0 0x00007fe6f6b5a680 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:81</font></div><div><font face="monospace, monospace">#1 0x00007fe6f795eea4 in poll (__timeout=-1, __nfds=2, __fds=0x7ffe4c0537b0) at /usr/include/bits/poll2.h:41</font></div><div><font face="monospace, monospace">#2 read_data (g=0x7fe6f8e04bd0, connv=0x7fe6f8df9100, bufv=<optimized out>, len=4) at conn-socket.c:162</font></div><div><font face="monospace, monospace">#3 0x00007fe6f7983f0c in recv_from_daemon (buf_rtn=0x7ffe4c053908, buf_rtn@entry=0x7ffe4c053658, size_rtn=0x7ffe4c0538e4, size_rtn@entry=0x7ffe4c053634, g=0x7fe6f8e04bd0, g@entry=0x7ffe4c053840) at proto.c:515</font></div><div><font face="monospace, monospace">#4 guestfs_int_recv_from_daemon (g=g@entry=0x7fe6f8e04bd0, size_rtn=size_rtn@entry=0x7ffe4c0538e4, buf_rtn=buf_rtn@entry=0x7ffe4c053908) at proto.c:611</font></div><div><font face="monospace, monospace">#5 0x00007fe6f797dae0 in launch_libvirt (g=0x7fe6f8e04bd0, datav=0x7fe6f8e04d90, libvirt_uri=<optimized out>) at launch-libvirt.c:573</font></div><div><font face="monospace, monospace">#6 0x00007fe6f7973b2b in guestfs_impl_launch (g=g@entry=0x7fe6f8e04bd0) at launch.c:93</font></div><div><font face="monospace, monospace">#7 0x00007fe6f790fb4d in <span style="background-color:rgb(255,255,255)"><font color="#ff0000">guestfs_launch </font></span>(g=0x7fe6f8e04bd0) at actions-3.c:142</font></div><div><font face="monospace, monospace">#8 0x00007fe6f86e7eda in run_launch (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds.c:13411</font></div><div><font face="monospace, monospace">#9 0x00007fe6f870276c in issue_command (cmd=0x7fe6f8df5170 "run", argv=argv@entry=0x7ffe4c053f98, pipecmd=0x0, rc_exit_on_error_flag=<optimized out>) at fish.c:1181</font></div><div><font face="monospace, monospace">#10 0x00007fe6f870334e in script (prompt=prompt@entry=1) at fish.c:733</font></div><div><font face="monospace, monospace">#11 0x00007fe6f86c7262 in interactive () at fish.c:630</font></div><div><font face="monospace, monospace">#12 main (argc=1, argv=0x7ffe4c0543f8) at fish.c:577</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div style="font-family:arial,helvetica,sans-serif">I tried export LIBGUESTFS_BACKEND=direct, and it is also stuck for more than 10 minutes, here is the call stack:</div></div><div><div><font face="arial, helvetica, sans-serif">0x00007f2dd9ac6ee0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81</font></div><div><font face="arial, helvetica, sans-serif">81<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)</font></div><div><font face="arial, helvetica, sans-serif">#0 0x00007f2dd9ac6ee0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81</font></div><div><font face="arial, helvetica, sans-serif">#1 0x00007f2dd9ac6d94 in __sleep (seconds=0, seconds@entry=2) at ../sysdeps/unix/sysv/linux/sleep.c:137</font></div><div><font face="arial, helvetica, sans-serif">#2 0x00007f2dda912089 in launch_direct (g=0x7f2ddc527bd0, datav=<optimized out>, arg=<optimized out>) at launch-direct.c:814</font></div><div><font face="arial, helvetica, sans-serif">#3 0x00007f2dda90eb2b in guestfs_impl_launch (g=g@entry=0x7f2ddc527bd0) at launch.c:93</font></div><div><font face="arial, helvetica, sans-serif">#4 0x00007f2dda8aab4d in guestfs_launch (g=0x7f2ddc527bd0) at actions-3.c:142</font></div><div><font face="arial, helvetica, sans-serif">#5 0x00007f2ddb682eda in run_launch (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at cmds.c:13411</font></div><div><font face="arial, helvetica, sans-serif">#6 0x00007f2ddb69d76c in issue_command (cmd=0x7f2ddc518170 "run", argv=argv@entry=0x7ffd9f69fe28, pipecmd=0x0, rc_exit_on_error_flag=<optimized out>) at fish.c:1181</font></div><div><font face="arial, helvetica, sans-serif">#7 0x00007f2ddb69e34e in script (prompt=prompt@entry=1) at fish.c:733</font></div><div><font face="arial, helvetica, sans-serif">#8 0x00007f2ddb662262 in interactive () at fish.c:630</font></div><div><font face="arial, helvetica, sans-serif">#9 main (argc=1, argv=0x7ffd9f6a0288) at fish.c:577</font></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">Has anyone seen this?</div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">Thanks. </div><div style="font-family:arial,helvetica,sans-serif">Allen</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-08-30 7:11 GMT+08:00 Baochuan Wu <span dir="ltr"><<a href="mailto:wildpointercs@gmail.com" target="_blank">wildpointercs@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Rich.<div>I have used libguestfs for several month. It worked perfectly before, the issue appears recently. I am trying <span style="font-size:14px">guestfs_set_backend (g, "direct").</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">Thanks,</span></div><div><span style="font-size:14px">Allen</span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-08-29 23:31 GMT+08:00 Richard W.M. Jones <span dir="ltr"><<a href="mailto:rjones@redhat.com" target="_blank">rjones@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Mon, Aug 29, 2016 at 11:19:04PM +0800, Baochuan Wu wrote:<br> > Thanks Rich for you quick reply. I enabled logs and the program stuck<br> > again, here is the call stack and log:<br> > Thread 1 (Thread 0x7fac58edc8c0 (LWP 1271)):<br> > #0 0x00007fac578fac20 in __poll_nocancel () from /lib64/libc.so.6<br> > #1 0x00007fac56df3c5a in virNetClientIOEventLoop () from<br> > /lib64/libvirt.so.0<br> > #2 0x00007fac56df441b in virNetClientSendInternal () from<br> > /lib64/libvirt.so.0<br> > #3 0x00007fac56df5843 in virNetClientSendWithReply () from<br> > /lib64/libvirt.so.0<br> > #4 0x00007fac56df6052 in virNetClientProgramCall () from<br> > /lib64/libvirt.so.0<br> > #5 0x00007fac56dcbfe2 in callFull.isra.2 () from /lib64/libvirt.so.0<br> > #6 0x00007fac56de213d in remoteDomainCreateXML () from /lib64/libvirt.so.0<br> > #7 0x00007fac56d82151 in virDomainCreateXML () from /lib64/libvirt.so.0<br> > #8 0x00007fac58acca50 in launch_libvirt () from /lib64/libguestfs.so.0<br> > #9 0x00007fac58ac2b2b in guestfs_impl_launch () from /lib64/libguestfs.so.0<br> > #10 0x00007fac58a5eba5 in guestfs_launch () from /lib64/libguestfs.so.0<br> > #11 0x00000000004117ca in main ()<br> </span>...<br> <span>> libguestfs: [62900ms] launch libvirt guest<br> <br> </span>The error is happening in libvirt's virDomainCreateXML call, called<br> from libguestfs here:<br> <br> <a href="https://github.com/libguestfs/libguestfs/blob/master/src/launch-libvirt.c#L600" rel="noreferrer" target="_blank">https://github.com/libguestfs/<wbr>libguestfs/blob/master/src/lau<wbr>nch-libvirt.c#L600</a><br> <br> Unfortunately libvirt isn't a simple C library. It will launch and<br> talk to a daemon (usually ``libvirt --timeout=30'' process, if you are<br> not running as root). Debugging into libvirtd is described here:<br> <br> <a href="http://libguestfs.org/guestfs-faq.1.html#debugging-libvirt" rel="noreferrer" target="_blank">http://libguestfs.org/guestfs-<wbr>faq.1.html#debugging-libvirt</a><br> <br> A workaround is to set LIBGUESTFS_BACKEND=direct [or use the<br> equivalent call ``guestfs_set_backend (g, "direct")''] which will<br> cause libguestfs to run qemu directly instead of going through<br> libvirt.<br> <br> <a href="http://libguestfs.org/guestfs.3.html#backend" rel="noreferrer" target="_blank">http://libguestfs.org/guestfs.<wbr>3.html#backend</a><br> <span><br> Rich.<br> <br> --<br> Richard Jones, Virtualization Group, Red Hat <a href="http://people.redhat.com/~rjones" rel="noreferrer" target="_blank">http://people.redhat.com/~rjon<wbr>es</a><br> Read my programming and virtualization blog: <a href="http://rwmj.wordpress.com" rel="noreferrer" target="_blank">http://rwmj.wordpress.com</a><br> </span>virt-df lists disk usage of guests without needing to install any<br> software inside the virtual machine. Supports Linux and Windows.<br> <a href="http://people.redhat.com/~rjones/virt-df/" rel="noreferrer" target="_blank">http://people.redhat.com/~rjon<wbr>es/virt-df/</a><br> </blockquote></div><br></div> </div></div></blockquote></div><br></div>