<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Title" content=""> <meta name="Keywords" content=""> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> I didn’t get the reply back in my email, but saw them on the thread, so here are the replies:<o:p></o:p> <o:p> </o:p> Additional debug shows the following in osa-dispatcher:<o:p></o:p> <o:p> </o:p> <-- <iq type='result' id='iq-request-0848dd-19'><query xmlns = 'jabber:iq:roster' ><item jid='osad-b6150a2570@SPACEWALKSERVER' subscription='to' /><item jid='osad-2bf4ba6a94@SPACEWALKSERVER' subscription='to' /><item jid='osad-7bad8b8265@SPACEWALKSERVER' subscription='to' /><item jid='osad-e1dda4dc96@SPACEWALKSERVER' subscription='to' /><item jid='osad-5763abe98f@SPACEWALKSERVER' subscription='to' /><item jid='osad-d042f17bd6@SPACEWALKSERVER' subscription='to' /><item jid='osad-aa8d364683@SPACEWALKSERVER' subscription='to' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='texv6hlicjon2s1ele1cst0zq28ux8krmimtz8bc'><query xmlns = 'jabber:iq:roster' ><item jid='osad-2bf4ba6a94@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> --> <iq type='get' id='iq-request-0848dd-20'><query xmlns = 'jabber:iq:roster' /></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='r00us18egaq83r8ttgorp9ibr54qdg35g3z957nl'><query xmlns = 'jabber:iq:roster' ><item jid='osad-d042f17bd6@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='rvh4518kr3dw1cj29f9ez20xo92ex6np14u762rx'><query xmlns = 'jabber:iq:roster' ><item jid='osad-7bad8b8265@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='65u7heaquj4tl5raftod0c22gw9nz1l56gdounfp'><query xmlns = 'jabber:iq:roster' ><item jid='osad-e1dda4dc96@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='jqxob49tlib3dzrn04townj807aoyihh8e5kjfd4'><query xmlns = 'jabber:iq:roster' ><item jid='osad-5763abe98f@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <presence from='osad-b6150a2570@SPACEWALKSERVER' type='subscribe' id='presence-088791-33' to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' /><o:p></o:p> <o:p> </o:p> --> <presence to='osad-b6150a2570@SPACEWALKSERVER' type='subscribed' id='presence-088791-33' /><o:p></o:p> <o:p> </o:p> <-- <iq type='result' id='iq-request-0848dd-20'><query xmlns = 'jabber:iq:roster' ><item jid='osad-b6150a2570@SPACEWALKSERVER' subscription='to' /><item jid='osad-2bf4ba6a94@SPACEWALKSERVER' subscription='both' /><item jid='osad-7bad8b8265@SPACEWALKSERVER' subscription='both' /><item jid='osad-e1dda4dc96@SPACEWALKSERVER' subscription='both' /><item jid='osad-5763abe98f@SPACEWALKSERVER' subscription='both' /><item jid='osad-d042f17bd6@SPACEWALKSERVER' subscription='both' /><item jid='osad-aa8d364683@SPACEWALKSERVER' subscription='to' /></query></iq><o:p></o:p> <o:p> </o:p> <-- <iq to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' type='set' id='l3ey7ycp9eljg1wmwxypgealw91g1cmng0lozyd8'><query xmlns = 'jabber:iq:roster' ><item jid='osad-b6150a2570@SPACEWALKSERVER' subscription='both' /></query></iq><o:p></o:p> <o:p> </o:p> --> <iq type='get' id='iq-request-0848dd-21'><query xmlns = 'jabber:iq:roster' /></iq><o:p></o:p> <o:p> </o:p> <-- <iq type='result' id='iq-request-0848dd-21'><query xmlns = 'jabber:iq:roster' ><item jid='osad-b6150a2570@SPACEWALKSERVER' subscription='both' /><item jid='osad-2bf4ba6a94@SPACEWALKSERVER' subscription='both' /><item jid='osad-7bad8b8265@SPACEWALKSERVER' subscription='both' /><item jid='osad-e1dda4dc96@SPACEWALKSERVER' subscription='both' /><item jid='osad-5763abe98f@SPACEWALKSERVER' subscription='both' /><item jid='osad-d042f17bd6@SPACEWALKSERVER' subscription='both' /><item jid='osad-aa8d364683@SPACEWALKSERVER' subscription='to' /></query></iq><o:p></o:p> <o:p> </o:p> --> <iq type='get' id='iq-request-0848dd-22'><query xmlns = 'jabber:iq:roster' /></iq><o:p></o:p> <o:p> </o:p> <-- <presence from='osad-6a13fc5ea7@SPACEWALKSERVER' type='subscribe' id='presence-6458c3-47' to='rhn-dispatcher-sat@SPACEWALKSERVER/superclient' /><o:p></o:p> <o:p> </o:p> --> <presence to='osad-6a13fc5ea7@SPACEWALKSERVER' type='subscribed' id='presence-6458c3-47' /><o:p></o:p> <o:p> </o:p> Spacewalk 18579 2017/09/18 17:49:47 -06:00: ('Error caught:',)<o:p></o:p> <o:p> </o:p> ERROR: unhandled exception occurred: (can't write str to text stream).<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> I don’t see any errors in /var/log/messages relating to jabberd, just the usual:<o:p></o:p> <o:p> </o:p> Sep 18 17:42:52 spacewalk2 jabberd/sm[18460]: [SPACEWALKSERVER] configured<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/sm[18460]: attempting connection to router at ::1, port=5347<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [::1, port=34632] connect<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [::1, port=34632] authenticated as jabberd@jabberd-router<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/sm[18460]: connection to router established<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [SPACEWALKSERVER] online (bound to ::1, port 34632)<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/c2s[18468]: modules search path: /usr/lib64/jabberd<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/c2s[18468]: loading 'db' authreg module<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/c2s[18468]: initialized auth module 'db'<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/c2s[18468]: starting up<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/c2s[18468]: process id is 18468, written to /var/lib/jabberd/pid/c2s.pid<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/sm[18460]: SPACEWALKSERVER ready for sessions<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/s2s[18476]: starting up (interval=3, queue=60, keepalive=0, idle=86400)<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/s2s[18476]: process id is 18476, written to /var/lib/jabberd/pid/s2s.pid<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/s2s[18476]: attempting connection to router at ::1, port=5347<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [::1, port=34634] connect<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [::1, port=34634] authenticated as jabberd@jabberd-router<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/s2s[18476]: connection to router established<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [s2s] set as default route<o:p></o:p> Sep 18 17:42:52 spacewalk2 jabberd/router[18452]: [s2s] online (bound to ::1, port 34634)<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: [SPACEWALKSERVER] configured; realm=, authreg=db, registration enabled, using PEM:/etc/pki/spacewalk/jabberd/server.pem<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: attempting connection to router at ::1, port=5347<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/router[18452]: [::1, port=34636] connect<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/router[18452]: [::1, port=34636] authenticated as jabberd@jabberd-router<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: connection to router established<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/router[18452]: [c2s] online (bound to ::1, port 34636)<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/s2s[18476]: [::, port=5269] listening for connections<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/s2s[18476]: ready for connections<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: [::, port=5222] listening for connections<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: ready for connections<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: [7] [::ffff:IPADDRESS, port=38292] connect<o:p></o:p> Sep 18 17:42:53 spacewalk2 jabberd/c2s[18468]: [7] [::ffff:IPADDRESS, port=38292] disconnect jid=unbound, packets: 0, bytes: 163<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/c2s[18468]: [7] [::ffff:192.168.0.85, port=58130] connect<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/c2s[18468]: [7] created user: user=osad-8041749890; realm=<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/c2s[18468]: [7] registration succeeded, requesting user creation: jid=osad-8041749890@SPACEWALKSERVER<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/sm[18460]: created user: jid=osad-8041749890@SPACEWALKSERVER<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/c2s[18468]: [7] traditional.digest authentication succeeded: osad-8041749890@/osad ::ffff:192.168.0.85:58130 TLS<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/c2s[18468]: [7] requesting session: jid=osad-8041749890@SPACEWALKSERVER/osad<o:p></o:p> Sep 18 17:42:54 spacewalk2 jabberd/sm[18460]: session started: jid=osad-8041749890@SPACEWALKSERVER/osad<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/c2s[18468]: [10] [::ffff:192.168.0.204, port=37242] connect<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/c2s[18468]: [10] created user: user=osad-cdee74d3cf; realm=<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/c2s[18468]: [10] registration succeeded, requesting user creation: jid=osad-cdee74d3cf@SPACEWALKSERVER<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/sm[18460]: created user: jid=osad-cdee74d3cf@SPACEWALKSERVER<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/c2s[18468]: [10] traditional.digest authentication succeeded: osad-cdee74d3cf@/osad ::ffff:192.168.0.204:37242 TLS<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/c2s[18468]: [10] requesting session: jid=osad-cdee74d3cf@SPACEWALKSERVER/osad<o:p></o:p> Sep 18 17:42:55 spacewalk2 jabberd/sm[18460]: session started: jid=osad-cdee74d3cf@SPACEWALKSERVER/osad<o:p></o:p> Sep 18 17:43:00 spacewalk2 jabberd/c2s[18468]: [11] [::ffff:192.168.0.149, port=33206] connect<o:p></o:p> Sep 18 17:43:00 spacewalk2 jabberd/c2s[18468]: [11] created user: user=osad-b4a12af2f9; realm=<o:p></o:p> Sep 18 17:43:00 spacewalk2 jabberd/c2s[18468]: [11] registration succeeded, requesting user creation: jid=osad-b4a12af2f9@SPACEWALKSERVER<o:p></o:p> Sep 18 17:43:00 spacewalk2 jabberd/sm[18460]: created user: jid=osad-b4a12af2f9@SPACEWALKSERVER<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: <spacewalk-list-bounces@redhat.com> on behalf of Jonathan Payne <Jonathan.Payne@franklincovey.com> Reply-To: "spacewalk-list@redhat.com" <spacewalk-list@redhat.com> Date: Monday, September 18, 2017 at 4:29 PM To: "spacewalk-list@redhat.com" <spacewalk-list@redhat.com> Subject: [Spacewalk-list] osa-dispatcher dies shortly after starting<o:p></o:p> </div> <div> <o:p> </o:p> </div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;cellpadding:0;cellspacing:0;display:table;border-collapse:seperate;float:none" id="msexchangesafetytips;9b4a17d0-9e03-40b3-e05a-08d4feebb451;R:en-US"> <tbody> <tr> <td style="background:#910A19;padding:5.25pt 1.5pt 5.25pt 1.5pt;valign:middle"></td> <td width="100%" style="width:100.0%;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt 11.25pt;word-wrap:break-word"> <div> This sender failed our fraud detection checks and may not be who they appear to be. Learn about <a href="http://aka.ms/LearnAboutSpoofing">spoofing</a><o:p></o:p> </div> </td> <td width="75" style="width:56.25pt;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt 3.75pt;word-wrap:break-word;align:left"> <a href="http://aka.ms/SafetyTipsFeedback">Feedback</a><o:p></o:p> </td> </tr> </tbody> </table> <div> Hoping for some assistance here, as I’ve been banging my head against the wall on this issue.<o:p></o:p> <o:p></o:p> I just upgraded from Spacewalk 2.2 to 2.6, and now osa-dispatcher is dying with no information on the error, and the status just saying the process is dead but the pid file still exists.<o:p></o:p> <o:p></o:p> LOG at Debug level 7:<o:p></o:p> 2017/09/18 11:33:25 -06:00 12686 0.0.0.0: rhnSQL/driver_postgresql.convert_named_query_params('Converting query for PostgreSQL: select * from rhnPushClient where jabber_id = :p1',)<o:p></o:p> 2017/09/18 11:33:25 -06:00 12686 0.0.0.0: rhnSQL/driver_postgresql.convert_named_query_params('New query: select * from rhnPushClient where jabber_id = %(p1)s',)<o:p></o:p> 2017/09/18 11:33:25 -06:00 12686 0.0.0.0: rhnSQL/driver_postgresql._execute_wrapper('Executing SQL: "select * from rhnPushClient where jabber_id = %(p1)s" with bind params: {p1: osad-6d4dfeb426@SPACEWALKSERVERNAME/osad}',)<o:p></o:p> 2017/09/18 11:33:25 -06:00 12686 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')<o:p></o:p> <o:p></o:p> <o:p></o:p> service osa-dispatcher status<o:p></o:p> osa-dispatcher dead but pid file exists<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> I have checked my jabberd xml files, and they appear to be setup correctly. One thing I found odd is that the server name shows up twice in the sm.xml. I changed the <id> under <sm> back to the default of sm, but that didnt' fix the issue.<o:p></o:p> <o:p></o:p> /etc/jabberd/sm.xml<o:p></o:p> <sm><o:p></o:p> <o:p></o:p> <id>SPACEWALKSERVERNAME</id><o:p></o:p> <o:p></o:p> <o:p></o:p> <local><o:p></o:p> <id>SPACEWALKSERVERNAME </id><o:p></o:p> <o:p></o:p> <id>localhost.localdomain</id><o:p></o:p> <o:p></o:p> <o:p></o:p> grep jabber /etc/rhn/rhn.conf<o:p></o:p> server.jabber_server = SPACEWALKSERVERNAME<o:p></o:p> osa-dispatcher.jabber_server = SPACEWALKSERVERNAME<o:p></o:p> <o:p></o:p> ll /etc/jabberd/*.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 25K Sep 17 20:20 /etc/jabberd/c2s.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 2.0K Jul 14 02:10 /etc/jabberd/router-filter.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 258 Jul 14 02:10 /etc/jabberd/router-users.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 6.9K Sep 17 20:20 /etc/jabberd/router.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 12K Sep 17 20:20 /etc/jabberd/s2s.xml<o:p></o:p> -rw-r-----. 1 jabber jabber 31K Sep 18 11:20 /etc/jabberd/sm.xml<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> I also checked the SSL certs to verify that they are correct and that the certificate matches the hostname in /etc/jabberd/c2s.xml. I have verified permissions, as well. Hostname is in the hosts file.<o:p></o:p> <o:p></o:p> md5sum /etc/pki/spacewalk/jabberd/server.pem /root/ssl-build/ SPACEWALKSERVERNAME server.pem<o:p></o:p> eda16d799aefba2acc5a5b4b278227ac /etc/pki/ SPACEWALKSERVERNAME jabberd/server.pem<o:p></o:p> eda16d799aefba2acc5a5b4b278227ac /root/ssl-build/ SPACEWALKSERVERNAME server.pem<o:p></o:p> <o:p></o:p> ll /etc/pki/spacewalk/jabberd/<o:p></o:p> -rw-------. 1 jabber jabber 7.2K Mar 4 2014 server.pem<o:p></o:p> <o:p></o:p> md5sum /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT<o:p></o:p> 4a3fa3d648a87aee70151dd6ee030c9b /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT<o:p></o:p> 4a3fa3d648a87aee70151dd6ee030c9b /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT<o:p></o:p> <o:p></o:p> grep CN= $(grep spacewalk.crt /etc/httpd/conf.d/ssl.conf | cut -f 2 -d' ') | grep Subject<o:p></o:p> OU= SPACEWALKSERVERNAME, CN= SPACEWALKSERVERNAME<o:p></o:p> <o:p></o:p> <o:p></o:p> grep require-starttls /etc/jabberd/c2s.xml | grep pemfile<o:p></o:p> <id require-starttls="false" pemfile="/etc/pki/spacewalk/jabberd/server.pem" realm="" register-enable="true">SPACEWALKSERVERNAME</id><o:p></o:p> <o:p></o:p> grep CN= /etc/pki/spacewalk/jabberd/server.pem | grep Subject<o:p></o:p> OU= SPACEWALKSERVERNAME, CN= SPACEWALKSERVERNAME<o:p></o:p> <o:p></o:p> grep "osa-dispatcher.osa_ssl_cert" /etc/rhn/rhn.conf<o:p></o:p> osa-dispatcher.osa_ssl_cert = /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT<o:p></o:p> <o:p></o:p> grep C2S_AS_ROOT /etc/sysconfig/jabberd <o:p></o:p> # set C2S_AS_ROOT to "yes" to start c2s as root<o:p></o:p> C2S_AS_ROOT="no"<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> I tried deleting the jabberd database and tables, but to no avail.<o:p></o:p> <o:p></o:p> rm -f /var/lib/jabberd/db/*<o:p></o:p> <o:p></o:p> spacewalk-sql -i<o:p></o:p> delete from rhnPushDispatcher;<o:p></o:p> delete from rhnPushClient;<o:p></o:p> \quit<o:p></o:p> <o:p></o:p> <o:p></o:p> I also tried re-configuring the jabberd service, to see if that would fix it, but it didn’t:<o:p></o:p> <o:p></o:p> service osa-dispatcher stop<o:p></o:p> service jabberd stop<o:p></o:p> <o:p></o:p> cp -arv /etc/jabberd /etc/jabberd.bak<o:p></o:p> cp -arv /var/lib/jabberd/ /var/lib/jabberd.bak<o:p></o:p> rm -rf /etc/jabberd/* /var/lib/jabberd/*<o:p></o:p> <o:p></o:p> yum reinstall jabberd -y<o:p></o:p> spacewalk-setup-jabberd<o:p></o:p> <o:p></o:p> grep $(hostname) /etc/jabberd/*xml<o:p></o:p> <o:p></o:p> service jabberd start<o:p></o:p> service osa-dispatcher start<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> I verified that ports are open:<o:p></o:p> <o:p></o:p> nmap -sT -p 5200-5400 localhost<o:p></o:p> <o:p></o:p> Nmap scan report for localhost (127.0.0.1)<o:p></o:p> PORT STATE SERVICE<o:p></o:p> 5222/tcp open xmpp-client<o:p></o:p> 5269/tcp open xmpp-server<o:p></o:p> 5347/tcp open unknown<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> I’ve run out of things to look at at this point. Any ideas?<o:p></o:p> </div> </div> </body> </html>