<div dir="ltr">Yeah unfortunately the osa-dispatcher code is unaware of the need for multiple dispatchers, which makes a multi-master setup for redundancy/failover difficult. Also, when you change the OSAD back end it will recreate the password and dirty hacks are necessary. Maybe this has been fixed in 2.6, however I had a very tough time trying to upgrade from 2.5 and had to revert, in my last attempt, and my priorities have shifted since then.<div><br></div><div>Just this past week I ran into an issue where removing some packages (vmware tools) also removed openjdk on our Spacewalk master, and I had to restore the master from backup. After that, getting all my clients back in online state with OSAD was a struggle... the dispatcher would crash at random times with little output due what I can only venture to guess is one client having an issue:</div><div><br></div><div><div>2017/03/30 14:56:57 -05:00 9475 <a href="http://0.0.0.0">0.0.0.0</a>: osad/jabber_lib.main('ERROR', (<type 'exceptions.NameError'>, NameError("global name 'jabber_id' is not defined",), <traceback object at 0x1b930e0>))</div></div><div><br></div><div>And when it crashed, it would take all my clients down and they would reregister, then crash again, until finally it got stable. This wouldn't be a big deal for a small environment, but at 6,000 systems plus, the traffic and load generated by OSAD clients doing this over and over again until the OSA dispatcher worked is embarrassing. I'm surprised it hasn't caused an outage yet...</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar 30, 2017 at 4:02 PM Camp, Neil (NIH/NCI) [C] <<a href="mailto:neil.camp@nih.gov">neil.camp@nih.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple" class="gmail_msg">
<div class="m_8584392146815608075WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">In the end I did end up moving to pgsql and it has really stabilized jabberd. I had to clear out the rhnpushdispatcher table a couple of times. I also got invalid password after moving
to pgsql. I found a post from <a href="https://www.redhat.com/archives/spacewalk-list/2016-August/msg00091.html" class="gmail_msg" target="_blank">
https://www.redhat.com/archives/spacewalk-list/2016-August/msg00091.html</a> which helped. It suggested<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">delete from authreg where username = 'rhn-dispatcher-sat';<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">delete from "roster-items" where "collection-owner" = 'rhn-dispatcher-sat@ourmaster1.fqdn';<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">delete from status where "collection-owner" = 'rhn-dispatcher-sat ourmaster1 fqdn';<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">delete from active where "collection-owner" = 'rhn-dispatcher-sat ourmaster1 fqdn';<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">From the jabberd database.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg">From: </span>
</b><span style="font-family:Calibri;color:black" class="gmail_msg"><<a href="mailto:spacewalk-list-bounces@redhat.com" class="gmail_msg" target="_blank">spacewalk-list-bounces@redhat.com</a>> on behalf of Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" class="gmail_msg" target="_blank">konrasko@gmail.com</a>><br class="gmail_msg">
<b class="gmail_msg">Reply-To: </b>"<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>" <<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>><br class="gmail_msg">
<b class="gmail_msg">Date: </b>Thursday, March 30, 2017 at 2:18 PM</span></p></div></div></div><div bgcolor="white" lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_8584392146815608075WordSection1 gmail_msg"><div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg"><br class="gmail_msg">
<b class="gmail_msg">To: </b>"<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>" <<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>><br class="gmail_msg">
<b class="gmail_msg">Subject: </b>Re: [Spacewalk-list] osa-dispatcher Error Caught<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p></div></div></div><div bgcolor="white" lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_8584392146815608075WordSection1 gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<table class="m_8584392146815608075MsoNormalTable gmail_msg" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;display:table;border-collapse:seperate" id="m_8584392146815608075msexchangesafetytips;e144803a-aca6-4294-945c-08d477996beb;R:en-US">
<tbody class="gmail_msg">
<tr class="gmail_msg">
<td style="background:#910a19;padding:5.25pt 1.5pt 5.25pt 1.5pt" class="gmail_msg"></td>
<td width="100%" style="width:100.0%;background:#fdf2f4;padding:5.25pt 3.75pt 5.25pt 11.25pt;word-wrap:break-word" class="gmail_msg">
<p class="MsoNormal gmail_msg">
<span style="font-size:9.0pt;font-family:"Segoe UI";color:#212121" class="gmail_msg">This message was identified as a <a href="http://aka.ms/LearnAboutPhishing" class="gmail_msg" target="_blank">phishing</a> scam.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</td>
<td width="75" style="width:56.25pt;background:#fdf2f4;padding:5.25pt 3.75pt 5.25pt 3.75pt;word-wrap:break-word" class="gmail_msg">
<p class="MsoNormal gmail_msg">
<span style="font-size:9.0pt;font-family:"Segoe UI";color:#212121" class="gmail_msg"><a href="http://aka.ms/SafetyTipsFeedback" class="gmail_msg" target="_blank">Feedback</a><u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
</td>
</tr>
</tbody>
</table>
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">I guess the main problem was caused by machines with login: <a href="mailto:osad-85cdcd1a3e@ncias-p1466-v.nci.nih.gov" class="gmail_msg" target="_blank"><span style="font-size:9.5pt" class="gmail_msg">osad-85cdcd1a3e</span></a>
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">I recommend to move from berkley to pgsql db<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">On Thu, Mar 30, 2017 at 9:49 AM, Camp, Neil (NIH/NCI) [C] <<a href="mailto:neil.camp@nih.gov" class="gmail_msg" target="_blank">neil.camp@nih.gov</a>> wrote:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in" class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg">I was able to connect to the database and select one row. I deleted the row in rhnpushdispatcher and stopped jabberd. I removed
/var/lib/jabberd/db/* and started jabberd. I waited for jabberd to start up and then started osa-dispatcher. It is staying up and I checked a host and it is showing as online for OSA status. Thank you for your help!</span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:Calibri" class="gmail_msg"> </span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-family:Calibri;color:black" class="gmail_msg">From:
</span></b><span style="font-family:Calibri;color:black" class="gmail_msg"><<a href="mailto:spacewalk-list-bounces@redhat.com" class="gmail_msg" target="_blank">spacewalk-list-bounces@redhat.com</a>> on behalf of Matt Moldvan <<a href="mailto:matt@moldvan.com" class="gmail_msg" target="_blank">matt@moldvan.com</a>><br class="gmail_msg">
<b class="gmail_msg">Reply-To: </b>"<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>" <<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>><br class="gmail_msg">
<b class="gmail_msg">Date: </b>Thursday, March 30, 2017 at 11:37 AM<br class="gmail_msg">
<b class="gmail_msg">To: </b>"<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>" <<a href="mailto:spacewalk-list@redhat.com" class="gmail_msg" target="_blank">spacewalk-list@redhat.com</a>><br class="gmail_msg">
<b class="gmail_msg">Subject: </b>Re: [Spacewalk-list] osa-dispatcher Error Caught</span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<table class="m_8584392146815608075MsoNormalTable gmail_msg" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;display:table;border-collapse:seperate" id="m_8584392146815608075m_-5439844986179069022msexchangesafetytips;23093ee8-5830-411e-dbfa-08d47783618b;R:en-US">
<tbody class="gmail_msg">
<tr class="gmail_msg">
<td style="background:#910a19;padding:5.25pt 1.5pt 5.25pt 1.5pt" class="gmail_msg"></td>
<td width="100%" style="width:100.0%;background:#fdf2f4;padding:5.25pt 3.75pt 5.25pt 11.25pt;word-wrap:break-word" class="gmail_msg">
<p class="MsoNormal gmail_msg">
<span style="font-size:9.0pt;font-family:"Segoe UI";color:#212121" class="gmail_msg">This message was identified as a <a href="http://aka.ms/LearnAboutPhishing" class="gmail_msg" target="_blank">phishing</a> scam.</span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</td>
<td width="75" style="width:56.25pt;background:#fdf2f4;padding:5.25pt 3.75pt 5.25pt 3.75pt;word-wrap:break-word" class="gmail_msg">
<p class="MsoNormal gmail_msg">
<span style="font-size:9.0pt;font-family:"Segoe UI";color:#212121" class="gmail_msg"><a href="http://aka.ms/SafetyTipsFeedback" class="gmail_msg" target="_blank">Feedback</a></span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</td>
</tr>
</tbody>
</table>
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Looks like osa-dispatcher is having trouble connecting to your database... have you tried running "spacewalk-sql -i" from your master (or the same system you're seeing that error
from) to get a n idea of the connectivity from that system to your database?<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Once you have that tested take a look at the rhnpushdispatcher table in the database. You can remove any entry there, osa-dispatcher will recreate it when you restart...<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">On Thu, Mar 30, 2017 at 10:47 AM Camp, Neil (NIH/NCI) [C] <<a href="mailto:neil.camp@nih.gov" class="gmail_msg" target="_blank">neil.camp@nih.gov</a>> wrote:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span class="m_8584392146815608075m-5439844986179069022gmailmsg gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">Hello,</span></span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span class="m_8584392146815608075m-5439844986179069022gmailmsg gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"> </span></span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span class="m_8584392146815608075m-5439844986179069022gmailmsg gmail_msg"><span style="font-size:11.0pt" class="gmail_msg">Osa-dispatcher starts, but dies after a few seconds. I have been digging through the archives and searching
but have not found a solution. </span></span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span class="m_8584392146815608075m-5439844986179069022gmailmsg gmail_msg"><span style="font-size:11.0pt" class="gmail_msg"> </span></span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">2017/03/30 10:35:25 -04:00 10622
<a href="http://0.0.0.0" class="gmail_msg" target="_blank">0.0.0.0</a>: osad/jabber_lib.subscribe_to_presence('Subscribed from', {})<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">2017/03/30 10:35:25 -04:00 10622
<a href="http://0.0.0.0" class="gmail_msg" target="_blank">0.0.0.0</a>: osad/jabber_lib.subscribe_to_presence('<a href="mailto:osad-85cdcd1a3e@ncias-p1466-v.nci.nih.gov" class="gmail_msg" target="_blank">osad-85cdcd1a3e@ncias-p1466-v.nci.nih.gov</a>',)<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">2017/03/30 10:35:25 -04:00 10622
<a href="http://0.0.0.0" class="gmail_msg" target="_blank">0.0.0.0</a>: rhnSQL/driver_postgresql._execute_wrapper('Executing SQL: "select * from rhnPushClient where jabber_id = %(p1)s" with bind params: {p1: osad-85cdcd1a3e@hostname/osad}',)<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">2017/03/30 10:35:26 -04:00 10622
<a href="http://0.0.0.0" class="gmail_msg" target="_blank">0.0.0.0</a>: osad/jabber_lib.main('ERROR', 'Error caught:')<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">I have turned debugging up to 5 for osa-dispatcher and have the last 4 lines posted above. Jabberd appears to be running correctly. I see connections coming in from clients. It
does have one error (SASL callback for non-existing host: spacewalk.fqdn). Does anyone have a suggestion as to what could be causing the error for osa-dispatcher?<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
</div>
<p class="MsoNormal gmail_msg">_______________________________________________<br class="gmail_msg">
Spacewalk-list mailing list<br class="gmail_msg">
<a href="mailto:Spacewalk-list@redhat.com" class="gmail_msg" target="_blank">Spacewalk-list@redhat.com</a><br class="gmail_msg">
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" class="gmail_msg" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal gmail_msg"><br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
Spacewalk-list mailing list<br class="gmail_msg">
<a href="mailto:Spacewalk-list@redhat.com" class="gmail_msg" target="_blank">Spacewalk-list@redhat.com</a><br class="gmail_msg">
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" class="gmail_msg" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</blockquote>
</div>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div>
</div></div>
_______________________________________________<br class="gmail_msg">
Spacewalk-list mailing list<br class="gmail_msg">
<a href="mailto:Spacewalk-list@redhat.com" class="gmail_msg" target="_blank">Spacewalk-list@redhat.com</a><br class="gmail_msg">
<a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" rel="noreferrer" class="gmail_msg" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a></blockquote></div>