[Spacewalk-list] osa-dispatcher still fails with Spacewalk 2.6

[Matt Moldvan]

Well, speak of the devil, osa-dispatcher crashed again today after I
botched the upgrade to 2.6.  It seems the dispatcher is crashing when it
detects an invalid client, maybe the jabber_id doesn't exist in the
rhnpushclient table?

2016/11/30 09:39:38 -05:00 20912 0.0.0.0: osad/jabber_lib.main('ERROR',
(<class 'osad.dispatcher_client.InvalidClientError'>, InvalidClientError('
osad-3a0bfed00d at spacewalk.prod.ch3.s.com/osad',), <traceback object at
0x14e3cf8>))

Seems that in the jabber_lib.py is where the exception is thrown, in main
somewhere at the very last except statement.  I didn't get any further than
that, though.

On Tue, Nov 29, 2016 at 11:12 AM Matt Moldvan <matt at moldvan.com> wrote:

> Ours has similar behavior on 2.5, crashing every so often.  I didn't find
> any useful information in any of the log files, even when setting debug
> modes relatively high.  I found the Python code that runs osa-dispatcher,
> and put a print statement in to watch when it crashes... unfortunately I
> didn't get much further than that, yet.
>
> As a quick and dirty workaround, I put a shell script in cron that
> restarts osa-dispatcher, and fixes the issue with it being coded to really
> only have one master dispatcher.  It's ugly, but it works... also, it
> assumes you are using SQLite as the jabberd back end, which seems to be
> more stable than Berkeley DB.
>
> #!/bin/bash
> PATH=/sbin:/usr/bin:/bin
> LOGFILE=/var/log/$(basename $0).out
> if [ ! -f "${LOGFILE}" ]; then
>         echo "osa-dispatcher and jabberd have been restarted 0 times since
> $(date); last restart was on " >> "${LOGFILE}"
> fi
> service jabberd status && service osa-dispatcher status
> if [ $? -ne 0 ]; then
>         service osa-dispatcher stop
>         service jabberd stop
>         echo "delete from rhnpushdispatcher where
> jabber_id='rhn-dispatcher-sat@$(uname -n)/superclient';" | spacewalk-sql
> -i
>         sqlite3 /var/lib/jabberd/db/sqlite.db <
> /usr/local/etc/fixjabber.sql
>         service jabberd start
>         service osa-dispatcher start
>         oldnum=$(cut -d ' ' -f7 ${LOGFILE})
>         newnum=$(expr $oldnum + 1)
>         sed -i "s/$oldnum times/$newnum times/g;s/last restart was
> on.*$/last restart was on $(date +'%F @ %H:%M:%S')" ${LOGFILE}
> fi
>
>
> On Tue, Nov 29, 2016 at 8:45 AM Ree, Jan-Albert van <J.A.v.Ree at marin.nl>
> wrote:
>
> A few things to debug this :
>
> - enable logging to file for s2s and c2s (in their config files under
> /etc/jabberd) and check those logfiles for issues
> - increase debug level of osa-dispatcher in /etc/rhn/rhn.conf (level 4-6
> usually is enough to start seeing the errors)
> - for us things became a lot more stable when we switched the jabberd
> backend from berkelyDB to sqlite
>
> Hope this helps...
> --
> Jan-Albert
>
>
>
> Jan-Albert van Ree  | Linux System Administrator | MARIN Support Group
> MARIN | T +31 317 49 35 48 <+31%20317%20493%20548> | mailto:
> J.A.v.Ree at marin.nl | http://www.marin.nl
>
> MARIN news:
> http://www.marin.nl/web/News/News-items/Symposium-Maritime-Design-Forum-1-december-2016.htm
>
> -----Original Message-----
> From: spacewalk-list-bounces at redhat.com [mailto:
> spacewalk-list-bounces at redhat.com] On Behalf Of Patrick Sachs
> Sent: Tuesday, 29 November, 2016 14:32
> To: spacewalk-list at redhat.com
> Subject: [Spacewalk-list] osa-dispatcher still fails with Spacewalk 2.6
>
> hi,
>
> i've updated Spacewalk 2.5 to 2.6. and the service osa-dispatcher still
> fails after a few minutes.
>
> System: CentOS 7
>
> osa-dispatcher.log::
>
> 2016/11/29 14:16:45 +02:00 17508 0.0.0.0: osad/jabber_lib.__init__
> 2016/11/29 14:16:45 +02:00 17508 0.0.0.0:
> osad/jabber_lib.setup_connection('Connected to jabber server', '
> spacewalk at teec.de')
> 2016/11/29 14:20:23 +02:00 17508 0.0.0.0: osad/jabber_lib.main('ERROR',
> 'Error caught:')
>
> not much helpful information :(
>
> Any ideas?
>
> Patrick
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20161130/0b8dfc9a/attachment.htm>