[Spacewalk-list] osa-dispatcher fails to start

Tue Mar 13 20:07:30 UTC 2018

On 03/13/18 20:30, Daryl Rose wrote:
> Alex,
> 
> 
> I just realized that I had more errors to look at.   I didn't check the error log prior to my last update.
> 
> 
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.check_cert('Loading cert', <X509Name object '/C=US/ST=VA/L=Herndon/O=Network Solutions L.L.C./CN=Network Solutions OV Server CA 2'>)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: rhnSQL/driver_postgresql.convert_named_query_params('Converting query for PostgreSQL: \n    select id, password\n      from rhnPushDispatcher\n     where jabber_id like :jabber_id\n    ',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: rhnSQL/driver_postgresql.convert_named_query_params('New query: \n    select id, password\n      from rhnPushDispatcher\n     where jabber_id like %(jabber_id)s\n    ',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: rhnSQL/driver_postgresql._execute_wrapper('Executing SQL: "\n    select id, password\n      from rhnPushDispatcher\n     where jabber_id like %(jabber_id)s\n    " with bind params: {jabber_id: rhn-dispatcher-sat%}',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.setup_connection('Connecting to', '<FQDN SW SERVER>')
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib._get_jabber_client
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib._get_jabber_client('Connecting to', '<FQDN SW SERVER>')
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.__init__
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.__init__
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.check_cert('Loading cert', <X509Name object '/C=US/ST=VA/L=Herndon/O=Network Solutions L.L.C./CN=Network Solutions OV Server CA 2'>)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('Attempting to connect',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process(300,)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process('before select(); timeout', 299.99999904632568)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process('select() returned',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib._auth_dispatch(<jabber.xmlstream.Node instance at 0x24bf950>,)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('Connected',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('Expecting features stanza, got:', <features><address xmlns = 'http://affinix.com/jabber/address' >::ffff:</address><auth xmlns = 'http://jabber.org/features/iq-auth'  /><register xmlns = 'http://jabber.org/features/iq-register'  /><starttls xmlns = 'urn:ietf:params:xml:ns:xmpp-tls' ><required /></starttls></features>)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('starttls node', <jabber.xmlstream.Node instance at 0x25389e0>)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process(None,)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process('before select(); timeout', None)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.process('select() returned',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib._auth_dispatch(<jabber.xmlstream.Node instance at 0x2538b90>,)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('Expecting proceed stanza, got:', <proceed />)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('Preparing for TLS handshake',)
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.connect('ERROR', 'Traceback caught:')
> 2018/03/13 14:24:59 -05:00 8164 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
> 
> Anything here that might be a clue as to what is going on?
> 
> Thanks
> 
> Daryl
> 

So your jabber cert seems to be /etc/pki/spacewalk/jabberd/server.pem

What does

openssl x509 -in /etc/pki/spacewalk/jabberd/server.pem -noout -enddate
-subject

tell?

Robert
> ________________________________
> From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Daryl Rose <darylrose at outlook.com>
> Sent: Tuesday, March 13, 2018 1:51 PM
> To: spacewalk-list at redhat.com
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> Alex,
> 
> 
> Sorry, can't/won't post /etc/hosts and /etc/sysconfig/network.  Trust me when I say that he server name is fully qualified.  Nothing in that regard has changed.  My suspicion was and still is that it's the cert.
> 
> 
> Looking back on this, I realize that I should have kept the self-signed certs in place, but for some reason I had to use a signed cert.  I used a doc that I found on Oracle explaining how to replace the self-signed certs with a CA signed cert.
> 
> 
> The cert expired in January.  I posted a question asking if I have to re-register all of our clients with a new, updated cert.  I was told no, that all I had to do was update the certificate for the WUI portion of SW.  However, I see that the jabber certificate, server.pem, did expire in January.
> 
> 
> I replaced the server.pem cert with the one that was generated when I updated the cert earlier this year.   I just now verified it against /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT and it came back okay.
> 
> 
> openssl verify -CAfile /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /etc/pki/spacewalk/jabberd/server.pem
> /etc/pki/spacewalk/jabberd/server.pem: OK
> 
> 
> However, I still get the same error when starting the osa-dispatcher.
> 
> 
> --> <?xml version='1.0' encoding='UTF-8'?><stream:stream to='<FQDN SW SERVER>' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
> 
> <-- <features><address xmlns = 'http://affinix.com/jabber/address' >::ffff:<IP></address><auth xmlns = 'http://jabber.org/features/iq-auth'  /><register xmlns = 'http://jabber.org/features/iq-register'  /><starttls xmlns = 'urn:ietf:params:xml:ns:xmpp-tls' ><required /></starttls></features>
> 
> <-- <proceed />
> 
> Spacewalk 6928 2018/03/13 13:41:37 -05:00: ('Traceback caught:',)
> Spacewalk 6928 2018/03/13 13:41:37 -05:00: ('Error caught:',)
> 
> ERROR: unhandled exception occurred: (can't write str to text stream).
> 
> 
> One question that I have, is what are these URL's?
> 
> http://affinix.com/jabber/address
> 
> http://jabber.org/features/iq-register
> 
> 
> I'm not knowledgeable in python, but it looks to me as if its registering with an external url and is using tls.   First of, why is jabber registering with an external url?  And, if it is, is it possible that the ca-certificates are out of date and need to be updated?
> 
> 
> Thanks
> 
> 
> Daryl
> 
> 
> 
> ________________________________
> From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Alexandru Raceanu <alex at capeno.com>
> Sent: Tuesday, March 13, 2018 12:18 PM
> To: spacewalk-list at redhat.com
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> Can you post the /etc/hosts and /etc/sysconfig/network ?
> if not... check this one: https://access.redhat.com/solutions/327903
> 
> /Alex
> 
> ________________________________
> From: "Daryl Rose" <darylrose at outlook.com>
> To: spacewalk-list at redhat.com
> Sent: Tuesday, March 13, 2018 4:22:41 PM
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> Yes, really, that was all.
> 
> 
> Here is the output from the very verbose osa-dispatcher start:
> 
> 
> /usr/sbin/osa-dispatcher -N -v -v -v -v -v -v -v
> --> <?xml version='1.0' encoding='UTF-8'?><stream:stream to='<FQDN SW SERVER>' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
> 
> <-- <features><address xmlns = 'http://affinix.com/jabber/address' >::ffff:10.255.0.6</address><auth xmlns = 'http://jabber.org/features/iq-auth'  /><register xmlns = 'http://jabber.org/features/iq-register'  /><starttls xmlns = 'urn:ietf:params:xml:ns:xmpp-tls' ><required /></starttls></features>
> 
> <-- <proceed />
> 
> Spacewalk 653 2018/03/13 10:18:35 -05:00: ('Traceback caught:',)
> Spacewalk 653 2018/03/13 10:18:35 -05:00: ('Error caught:',)
> 
> ERROR: unhandled exception occurred: (can't write str to text stream).
> 
> Thank you
> 
> 
> Daryl
> 
> ________________________________
> From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> on behalf of Paschedag, Robert <paschedag.netlution at swr.de>
> Sent: Tuesday, March 13, 2018 8:47 AM
> To: spacewalk-list at redhat.com
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> Is that really all??
> 
> 
> 
> With this information only, it is impossible to help.
> 
> 
> 
> You can try to run osa-dispatcher manually
> 
> 
> 
> Stop it
> 
> 
> 
> /etc/init.d/osa-dispatcher stop
> 
> 
> 
> Run it manually
> 
> 
> 
> /usr/sbin/osa-dispatcher -N -v -v -v -v -v -v -v
> 
> 
> 
> and post errors you get.
> 
> 
> 
> Robert
> 
> 
> 
> 
> 
> Von: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> Im Auftrag von Daryl Rose
> Gesendet: Dienstag, 13. März 2018 14:14
> An: spacewalk-list at redhat.com
> Betreff: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> 
> Here are the requested entries.
> 
> 
> 
> 2018/03/13 08:11:08 -05:00 45604 0.0.0.0: osad/jabber_lib.__init__
> 
> 2018/03/13 08:11:08 -05:00 45604 0.0.0.0: osad/jabber_lib.connect('ERROR', 'Traceback caught:')
> 
> 2018/03/13 08:11:08 -05:00 45604 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
> 
> 
> 
> Thank you.
> 
> 
> 
> Daryl
> 
> 
> 
> 
> 
> ________________________________
> 
> From: spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com> <spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com>> on behalf of Alexandru Raceanu <alex at capeno.com<mailto:alex at capeno.com>>
> Sent: Monday, March 12, 2018 12:52 PM
> To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> 
> Can you provide the entries after a "/etc/init.d/osa-dispatcher restart" from /var/log/rhn/osa-dispatcher.log ?
> 
> 
> 
> /Alex
> 
> 
> 
> 
> 
> ________________________________
> 
> From: "Daryl Rose" <darylrose at outlook.com<mailto:darylrose at outlook.com>>
> To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
> Sent: Monday, March 12, 2018 4:28:18 PM
> Subject: Re: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> 
> I'm sorry, I just realized that I should have provided more information.
> 
> 
> 
>   1.  This is a RHEL 6.8 server
>   2.  SW v2.6.
> 
> 
> 
> Also, we use a signed cert and the certificate expired January.  I inquired on this list and I was told that I only needed to update the cert for the website portion of the cert that they cert used to communicate between the systems did not need to be changed.
> 
> 
> 
> I'm guessing its a cert issue from some of the research that I did, but I'm not sure what I should update, or if I really need to.
> 
> 
> 
> Thanks
> 
> 
> 
> Daryl
> 
> 
> 
> ________________________________
> 
> From: spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com> <spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com>> on behalf of Daryl Rose <darylrose at outlook.com<mailto:darylrose at outlook.com>>
> Sent: Monday, March 12, 2018 10:19 AM
> To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
> Subject: [Spacewalk-list] osa-dispatcher fails to start
> 
> 
> 
> osa-dispatcher is down and won't start.   I get the following error when trying to start it:
> 
> 
> 
> Starting osa-dispatcher: Spacewalk 43238 2018/03/12 10:17:13 -05:00: ('Traceback caught:',)
> 
> Spacewalk 43238 2018/03/12 10:17:13 -05:00: ('Error caught:',)
> 
> 
> 
> ERROR: unhandled exception occurred: (can't write str to text stream).
> 
>                                                            [FAILED]
> 
> 
> 
> Any suggestions?
> 
> 
> 
> Thanks
> 
> 
> 
> Daryl
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 
> 
> 
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
> 