[Libguestfs] Using guestfs.kill_subprocess()

[Richard Su]

On 11/04/2011 01:29 AM, Richard W.M. Jones wrote:
> On Thu, Nov 03, 2011 at 09:41:40PM -0400, Chris Lalancette wrote:
>> Hey Rich,
>>       In the past I know you have mentioned that using
>> guestfs.kill_subprocess() isn't strictly necessary after you are finished
>> using libguestfs.  I believe you said that in the python bindings, that
>> step would happen automatically when the object goes out of scope and the
>> __del__ method is called.  However, since it never seemed to harm anything
>> and I wanted to be extra safe, I always called guestfs.kill_subprocess() in
>> Oz, without apparent harm.
> Python bindings now have an explicit close call:
>
> 8c5bdc3e12947580e91c018b71adf9ad3128bb75
> http://bugzilla.redhat.com/717786
>
>>       However, Richard Su has been investigating an issue where
>> guestfs.kill_subprocess() actually does cause harm.  I believe the
>> situation is that running Oz (and hence libguestfs) simultaneously from two
>> threads in the same process causes the entire process to hang.  If we
>> remove the guestfs.kill_subprocess(), the problem goes away.  rwsu, can you
>> please fill in a bit more detail here, as I'm not intimately familiar with
>> the issue?  Things like host OS, libguestfs version, etc etc would be
>> helpful.  Also confirmation that the problem is indeed in the
>> multi-threaded environment of the image factory like I think it is.
>>       rjones, do you have any ideas here?  I'm going to merge rwsu's patch
>> to remove the call to kill_subprocess(), but I thought it would be worth a
>> little more digging into why it was happening.
> You could be hitting this bug:
> http://bugzilla.redhat.com/684980
>
> To debug this I need to see:
>
>   - version of libguestfs
>
>   - platform that you're running it on
>
>   - full output with LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 set
>
> Rich.
>

The problem I'm running into is when I start two image builds, and thus 
two guest installs, they run but eventually hang when killing guestfs 
subprocess is called. Here's the last lines of the imagefactory logs.

2011-11-04 12:44:51,299 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Step 3: Open up the firewall
2011-11-04 12:44:51,299 INFO oz.Guest.FedoraGuest pid(23842) Message: 
Collection Setup
2011-11-04 12:44:51,300 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Step 4: Guest announcement
2011-11-04 12:44:51,320 INFO oz.Guest.FedoraGuest pid(23842) Message: 
Setting up guestfs handle for tmpl1-7dd71f1d-9bc5-43d1-a685-0d8397eb562c
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Step 5: Set SELinux to permissive mode
2011-11-04 12:44:51,337 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Adding disk image 
/var/lib/imagefactory/images/rhevm-image-7dd71f1d-9bc5-43d1-a685-0d8397eb562c.dsk
2011-11-04 12:44:51,356 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Launching guestfs
2011-11-04 12:44:53,974 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Inspecting guest OS
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Getting mountpoints
2011-11-04 12:44:55,049 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Root device: /dev/VolGroup00/LogVol00
2011-11-04 12:44:55,152 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Step 1: Uploading ssh keys
2011-11-04 12:44:55,156 INFO oz.Guest.FedoraGuest pid(23842) Message: 
Cleaning up guestfs handle for tmpl1-207802cf-4f02-4faa-8b28-1d896fd44ca9
2011-11-04 12:44:55,168 INFO oz.Guest.FedoraGuest pid(23842) Message: 
Generating new openssh key
2011-11-04 12:44:55,169 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Syncing
2011-11-04 12:44:55,193 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Unmounting all
2011-11-04 12:44:55,195 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Step 2: setup sshd
2011-11-04 12:44:55,204 DEBUG oz.Guest.FedoraGuest pid(23842) Message: 
Killing guestfs subprocess

One of the qemu process dies but the other one remains up but doesn't move.

I'm running this on RHEL 6.1 with libguestfs-1.7.17-17.el6.x86_64.

I'm having trouble producing trace output. I tried setting 
LIBGUESTFS_TRACE and LIBGUESTFS_DEBUG and since this is through python, 
I also tried calling set_trace and set_verbose in this way in 
/usr/lib/python2.6/site-packages/oz/Guest.py:

     g = guestfs.GuestFS()
     g.set_trace(1)
     g.set_verbose(1)

Any suggestions here?