[Spacewalk-list] OSAD randomly becomes offline

[Piotr Dzionek]

Hi,

I noticed that many systems registered with my Spacewalk 2.6 server 
become offline from time to time. This means that I cannot update them 
in real-time. They sometimes stop being offline if I perform some 
restarts including whole spacewalk server, but it doesn't always solve 
this problem. I dig into OSAD logs on some of the clients that are 
offline and I found out that they communicate with the main jabberd 
server/osa dispatcher, but the they are not being set as online.

I only noticed that on the faulty nodes in the callback response there 
is subscription='to' not the subscription='both', like it is on the 
working nodes. I read somewhere that it should to be set to "both", but 
I am not an expert :(

/2017-07-12 12:00:14 jabber_lib.setup_connection: Connected to jabber 
server gt-prd-spacewalk-srv01.internal
2017-07-12 12:00:14 jabber_lib.push_to_background: Pushing process into 
background
2017-07-12 12:00:14 osad_client.start: osad-6d801d3a52 
d7e9650f284e2a58eaa8 osad
2017-07-12 12:00:14 jabber_lib.auth: osad-6d801d3a52 
d7e9650f284e2a58eaa8 osad 1
2017-07-12 12:00:14 jabber_lib.auth: Sending auth request <iq type='get' 
id='auth-get-d76052-0'><query xmlns = 'jabber:iq:auth' 
 ><username>osad-6d801d3a52</username></query></iq>
2017-07-12 12:00:14 jabber_lib.process: 59.999984026
2017-07-12 12:00:14 jabber_lib.dispatch: Unhandled stanza <iq 
type='result' id='auth-get-d76052-0'><query xmlns = 'jabber:iq:auth' 
 ><username>osad-6d801d3a52</username><resource /><password /><digest 
/></query></iq>
2017-07-12 12:00:14 jabber_lib.auth: Auth response <iq type='result' 
id='auth-get-d76052-0'><query xmlns = 'jabber:iq:auth' 
 ><username>osad-6d801d3a52</username><resource /><password /><digest 
/></query></iq>
2017-07-12 12:00:14 jabber_lib.auth: Sending auth info <iq type='set' 
id='auth-set-d76052-1'><query xmlns = 'jabber:iq:auth' 
 ><username>osad-6d801d3a52</username><resource>osad</resource><digest>b06f017d6001bef84eff8014b248987becaf1d2d</digest></query></
iq>
2017-07-12 12:00:14 jabber_lib.process: 299.999989986
2017-07-12 12:00:14 jabber_lib.dispatch: Unhandled stanza <iq 
type='result' id='auth-set-d76052-1' />
2017-07-12 12:00:14 jabber_lib.auth: Authenticated
2017-07-12 12:00:14 jabber_lib.register_callback: <bound method 
Client._roster_callback of <osad.osad_client.Client object at 0xfd6dd0>> 
iq None None None None
2017-07-12 12:00:14 jabber_lib.process: None
2017-07-12 12:00:14 jabber_lib._roster_callback: Updating the roster <iq 
type='result' id='iq-request-d76052-2'><query xmlns = 'jabber:iq:roster' 
 ><item jid='rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal' 
subscription='to' /></query></iq>
2017-07-12 12:00:14 jabber_lib.register_callback: <bound method 
Client._presence_callback of <osad.osad_client.Client object at 
0xfd6dd0>> presence None None None None
2017-07-12 12:00:14 jabber_lib.register_callback: <bound method 
Client._message_callback of <osad.osad_client.Client object at 
0xfd6dd0>> message None None None None
2017-07-12 12:00:14 jabber_lib.register_callback: <bound method 
Runner._error_callback of <osad.osad.Runner instance at 0x7f013cae9830>> 
error None None None None
2017-07-12 12:00:14 osad.fix_connection: Time drift 0
2017-07-12 12:00:14 osad.fix_connection: Client name 42666bad562d6566
2017-07-12 12:00:14 osad.fix_connection: Shared key 
715211881f09a2ab623434d16a031a9a054d316f
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed to 
{'rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal': {'jid': 
'rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal', 'subscription': 
u'to'}}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed both {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed none {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed from {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed to
2017-07-12 12:00:14 jabber_lib.send_presence: None None
2017-07-12 12:00:14 jabber_lib.process_forever:
2017-07-12 12:00:14 jabber_lib.process: 180
2017-07-12 12:00:14 jabber_lib._presence_callback: 
osad-6d801d3a52 at gt-prd-spacewalk-srv01.internal/osad 
rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal/superclient None
2017-07-12 12:00:14 jabber_lib._presence_callback: Node is available 
rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal/superclient None
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed to 
{'rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal': {'jid': 
'rhn-dispatcher-sat at gt-prd-spacewalk-srv01.internal', 'subscription': 
u'to'}}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed both {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed none {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed from {}
2017-07-12 12:00:14 jabber_lib.subscribe_to_presence: Subscribed to/

As you can see this request is sent repeatedly almost every 10 seconds 
and host don't become online.  Also I can see that the there are some 
logs regarding this client in osa-dispatcher logfile.

What can be an issue here ? Is Jabber blocking some types of connections 
or maybe there is an issue with sqllite db ? or even with spacewalk db ? 
Because this problem is not persistent, like I said with some restarts 
some of the nodes become online.

Kind regards,
Piotr Dzionek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20170712/2f127332/attachment.htm>