[Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

Tue Mar 11 14:50:09 UTC 2014

Hmm, interesting thought about alternate jabber servers... I'd be very interested to hear how that goes. If I can't resolve this with the current config, I'll try that route also.

I looked at the change-logs for the 2.1 distribution; didn't see anything directly related, but it is worth a try.

Best Regards,

Whit

--
Whitney "Whit" Latta
Sr. Systems Engineer
Linux - Open Systems Engineering
whitney_latta at homedepot.com<mailto:whitney_latta at homedepot.com>
Office: (770) 433-8211 ext:81278
Mobile: (678) 231-2049

--
"We've done the impossible, and that makes us mighty." ―Malcolm Reynolds
--

From: Michael Dorman [mailto:mdorman at godaddy.com]
Sent: Tuesday, March 11, 2014 10:38 AM
To: spacewalk-list at redhat.com; Latta, Whitney
Subject: Re: [Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

We've seen this same behavior, too, and have not been able to narrow down a good solution.  One idea we had was to replace the out of the box Jabber server with the one from jabber.org or ejabberd, to see if that would improve things.  But we haven't been able to experiment with that yet.

We're just now starting to look at 2.1.  I haven't reviewed the change log or anything to see if there are any OSAD improvements there.

It is frustrating, though, because we rely on the remote command execution functions of OSAD to do a lot of stuff.  And when there's a handful of servers that don't pick up the action right away, it's a real pain.  So definitely would be curious to know about any other info you find out.

Mike

From: "WHITNEY_LATTA at homedepot.com<mailto:WHITNEY_LATTA at homedepot.com>" <WHITNEY_LATTA at homedepot.com<mailto:WHITNEY_LATTA at homedepot.com>>
Reply-To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Date: Tuesday, March 11, 2014 7:51 AM
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: [Spacewalk-list] osad fails to pick up queued actions, when client is behind proxy server...

Hello,

I've pretty much run out of ideas to troubleshoot this issue further; I am hoping someone can suggest some more in-depth troubleshooting of the jabberd communications in Spacewalk v2.0. Googling the symptoms, I found several others with the same issue, but no clear resolution... everything that has previously been suggested has come up empty.

The issue we're seeing is that osad and osa-dispatcher on the client and server side appear to connect to the proxy-server, but osa ping from the Spacewalk gui does nothing, and any actions queued up to the client are not picked up until the next rhn_check is exec'ed manually or from rhnsd.  Running rhn_check works fine; in fact everything else in Spacewalk works fine, except the osa-dispatcher push to the clients.

So far, we've tried shutting down all the jabberd and osa services on the Spacewalk, proxy, and client servers; then, clear the /var/lib/jabberd/db/* files on the Spacewalk and proxy servers; then, remove the osad-auth.conf file on the proxy and client... restart everything in order (Spacewalk, proxy, client)... the issue persists... ping to the proxy server returns fine, but not for the end client. If the client is set to connect to the Spacewalk server, it works, but not through the proxy server.

Checked the ssl certs and checksums/permissions appear correct. The OU and CN point to the spacewalk server hostname. I even tried spacewalk-hostname-rename to no avail.

The one gotcha in this is that we're going through a citrix netscaler load-balancer in front of the Spacewalk server (actually a pair, in an active-standby configuration, so only 1 is up at any one time). If I eliminate the LB from the configuration and setup with the original FQDN hostnames, the jabberd communications through osad appears to work fine from the client through the proxy server to Spacewalk. It's just frustrating that the connections appear to be made, but no communication is going through in this configuration, and no errors or other messages can be found to identify where it is failing... It appears to be ssl that is failing, but how to isolate the root cause?

Is there some way to generate more debug output specifically from the connection, and in particular the ssl interactions? I set debug levels in osad to 7 and in rhn.conf to 5, and it generates a lot of output, but I can't get any details on what is happening between the client, through the proxy server to Spacewalk... or I'm missing something.

Curiously, I do see that osa-dispatcher doesn't seem to find any servers to ping:

osad/osa_dispatcher.process_once('Clients to be pinged:', None)

Looking at the rhnpushclient table, it appears the ping was entered, but the LAST_PING_TIME field is never cleared (until I restart the services):

SQL> select ID,JABBER_ID,LAST_PING_TIME from rhnpushclient;

        ID
----------
JABBER_ID
--------------------------------------------------------------------------------
LAST_PING_TIME
---------------------------------------------------------------------------
        11
osad-905526b856@<proxy-server-FQDN>/osad
11-MAR-14 09.38.56.615000 AM

         9
osad-85d7c101c5@<spacewalk-server-FQDN>/osad

Any help would be greatly appreciated.

Best Regards,

Whit

--
Whitney "Whit" Latta
Sr. Systems Engineer
Linux - Open Systems Engineering

--
"We've done the impossible, and that makes us mighty." ―Malcolm Reynolds
--

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20140311/e70431db/attachment.htm>