[Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

[WHITNEY_LATTA at homedepot.com]

Hello,

I've pretty much run out of ideas to troubleshoot this issue further; I am hoping someone can suggest some more in-depth troubleshooting of the jabberd communications in Spacewalk v2.0. Googling the symptoms, I found several others with the same issue, but no clear resolution... everything that has previously been suggested has come up empty.

The issue we're seeing is that osad and osa-dispatcher on the client and server side appear to connect to the proxy-server, but osa ping from the Spacewalk gui does nothing, and any actions queued up to the client are not picked up until the next rhn_check is exec'ed manually or from rhnsd.  Running rhn_check works fine; in fact everything else in Spacewalk works fine, except the osa-dispatcher push to the clients.

So far, we've tried shutting down all the jabberd and osa services on the Spacewalk, proxy, and client servers; then, clear the /var/lib/jabberd/db/* files on the Spacewalk and proxy servers; then, remove the osad-auth.conf file on the proxy and client... restart everything in order (Spacewalk, proxy, client)... the issue persists... ping to the proxy server returns fine, but not for the end client. If the client is set to connect to the Spacewalk server, it works, but not through the proxy server.

Checked the ssl certs and checksums/permissions appear correct. The OU and CN point to the spacewalk server hostname. I even tried spacewalk-hostname-rename to no avail.

The one gotcha in this is that we're going through a citrix netscaler load-balancer in front of the Spacewalk server (actually a pair, in an active-standby configuration, so only 1 is up at any one time). If I eliminate the LB from the configuration and setup with the original FQDN hostnames, the jabberd communications through osad appears to work fine from the client through the proxy server to Spacewalk. It's just frustrating that the connections appear to be made, but no communication is going through in this configuration, and no errors or other messages can be found to identify where it is failing... It appears to be ssl that is failing, but how to isolate the root cause?

Is there some way to generate more debug output specifically from the connection, and in particular the ssl interactions? I set debug levels in osad to 7 and in rhn.conf to 5, and it generates a lot of output, but I can't get any details on what is happening between the client, through the proxy server to Spacewalk... or I'm missing something.

Curiously, I do see that osa-dispatcher doesn't seem to find any servers to ping:

osad/osa_dispatcher.process_once('Clients to be pinged:', None)

Looking at the rhnpushclient table, it appears the ping was entered, but the LAST_PING_TIME field is never cleared (until I restart the services):


SQL> select ID,JABBER_ID,LAST_PING_TIME from rhnpushclient;

        ID
----------
JABBER_ID
--------------------------------------------------------------------------------
LAST_PING_TIME
---------------------------------------------------------------------------
        11
osad-905526b856@<proxy-server-FQDN>/osad
11-MAR-14 09.38.56.615000 AM

         9
osad-85d7c101c5@<spacewalk-server-FQDN>/osad

Any help would be greatly appreciated.


Best Regards,

Whit

--
Whitney "Whit" Latta
Sr. Systems Engineer
Linux - Open Systems Engineering

--
"We've done the impossible, and that makes us mighty." ―Malcolm Reynolds
--


________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20140311/6e6f7cd1/attachment.htm>