[Spacewalk-list] rhn_check hangs

[Gerald Vogt]

Hi!

Since the day before yesterday some of my spacewalk clients have
troubles running a queued comparison of config files.

When I run "rhn_check -v -v" I can see that the client picks up the
action and even writes

  D: Sending back response(0, 'Files successfully diffed',

After that it hangs and eventually stops with:

> Could not submit to <RetryServer for spacewalk.dkrz.de/XMLRPC>.
> Possible networking problem?

Any attempt after that will hang immediately and fail.

> # rhn_check -v -v
> Could not retrieve action from <RetryServer for spacewalk.dkrz.de/XMLRPC>.
> Possible networking problem?

On the spacewalk server the rhn_server_xmlrpc.log shows this:

> 2012/10/04 09:41:11 +02:00 22595 136.172.30.253: xmlrpc/queue.get(1000010027, 2, 'checkins enabled')
> 2012/10/04 09:41:16 +02:00 22601 136.172.30.253: xmlrpc/up2date.login(1000010027,)
> 2012/10/04 09:41:16 +02:00 22598 136.172.30.253: xmlrpc/queue.submit(1000010027, 2965, 0)

The httpd process connected to the client hangs, too. I can see the
process in the apache server status as "Sending Reply" even though the
connection has been closed by the client.

If I restart the httpd process on the server, the client will pick up
the action from the queue again but will hang and fail as before. So it
seems to be a problem on the server hanging while processing the results
of the comparison...

I have tried to find additional logs which may show more or get some
debug logs active for the XMLRPC processing but haven't found how to do
that.

The apache error_log shows this when the client submits the results:

> Modules/gcmodule.c:348: visit_decref: Assertion "gc->gc.gc_refs != 0" failed.
> refcount was too small

Any ideas how to fix this problem or how to troubleshoot it?
How do I enable debug logging for the scripts running on the XMLRPC URL?

Thanks!

Cheers,

Gerald