[Spacewalk-list] OSA Not Picking Up Actions - Most of the Time

[Wojtak, Greg (Superfly)]

I've been having troubles on and off with OSA picking up actions for a long time now (since the middle of 2011).  It's been broken completely for months and I had given up on it for a while so I could focus on other projects.  Recently, I updated the spacewalk client on one of my Cent 5 servers to the packages from the 1.6 repository (everything else is 1.5).  That client started picking up actions.  I upgraded a Cent 6 client and it started picking up actions.  Now, we're back to everything not working again.  I've tried stopping OSA clients, osa-dispatcher, and jabberd and cleaning out /var/lib/jabber/db, but still no scheduled actions get picked up.  I'm not sure what's changing that causes this to break.  At one point when I first set this up, all clients were picking up tasks normally (even those behind our proxy).

I upped the verbosity of everything OSA related I could think of (jabber, osad, osa-dispatcher) and I noticed in the osa-dispatcher.log that I keep getting the following message repeated every 15 seconds or so:

2012/01/24 14:13:39 -04:00 10143 0.0.0.0: osad/jabber_lib._roster_callback('Updating the roster', <iq type='result' id='iq-request-1485e3-1362'><query xmlns = 'jabber:iq:roster' ><item jid='osad-6dd99be55c at spacewalk' subscription='both' /><item jid='osad-85a280d1a5 at spacewalk' subscription='both' /><item jid='osad-25ed3808b1 at spacewalk' subscription='both' /><item jid='osad-b3b8a0898c at spacewalk' subscription='both' /><item jid='osad-b902405a31 at spacewalk' subscription='both' /><item jid='osad-fe3f0f59f6 at spacewalk' subscription='both' /><item jid='osad-10b42d611a at spacewalk' subscription='both' /><item jid='osad-f0f388c885 at spacewalk' subscription='both' /><item jid='osad-6021c959dd at spacewalk' subscription='both' /><item jid='osad-28dc0ddee4 at spacewalk' subscription='both' /><item jid='osad-403d7f1572 at spacewalk' subscription='both' /><item jid='osad-c294e8be55 at spacewalk' subscription='both' /><item jid='osad-bdd7889037 at spacewalk' subscription='both' /><item jid='osad-6398479353 at spacewalk' subscription='both' /><item jid='osad-c182dc85c6 at spacewalk' subscription='both' /><item jid='osad-11a37bc699 at spacewalk' subscription='both' /><item jid='osad-77dbbcccd5 at spacewalk' subscription='both' /><item jid='osad-c67db0d167 at spacewalk' subscription='both' /><item jid='osad-bf92fac08a at spacewalk' subscription='both' /><item jid='osad-bfbff17037 at spacewalk' subscription='both' /><item jid='osad-18812f1a6b at spacewalk' subscription='both' /><item jid='osad-e649e75e71 at spacewalk' subscription='both' /><item jid='osad-2a7e50b55f at spacewalk' subscription='both' /><item jid='osad-688025edeb at spacewalk' subscription='both' /><item jid='osad-4b6e8f533a at spacewalk' subscription='both' /><item jid='osad-0451103b21 at spacewalk' subscription='both' /><item jid='osad-b011b1c0d2 at spacewalk' subscription='both' /><item jid='osad-451aaaee64 at spacewalk' subscription='both' /><item jid='osad-1b0a719359 at spacewalk' subscription='both' /><item jid='osad-194a9a90a0 at spacewalk' subscription='both' /><item jid='osad-67b967a5ec at spacewalk' subscription='both' /><item jid='osad-9eebbd9684 at spacewalk' subscription='both' /><item jid='osad-af9f4b0a93 at spacewalk' subscription='both' /><item jid='osad-2dee90ef72 at spacewalk' subscription='both' /><item jid='osad-c4dabbab4a at spacewalk' subscription='both' /><item jid='osad-82c3fd1196 at spacewalk' subscription='both' /><item jid='osad-f6336873b9 at spacewalk' subscription='both' /><item jid='osad-08e127bbe1 at spacewalk' subscription='both' /><item jid='osad-981cef0a59 at spacewalk' subscription='both' /><item jid='osad-2299f285cb at spacewalk' subscription='both' /><item jid='osad-e5bb60fae6 at spacewalk' subscription='both' /><item jid='osad-ceee60b35a at spacewalk' subscription='both' /><item jid='osad-4ab5858f26 at spacewalk' subscription='both' /><item jid='osad-c5d8b414cf at spacewalk' subscription='both' /><item jid='osad-ed5c434fb0 at spacewalk' subscription='both' /><item jid='osad-395b66990a at spacewalk' subscription='both' /><item jid='osad-4d777a57e3 at spacewalk' subscription='both' /><item jid='osad-dbb78ff69a at spacewalk' subscription='both' /><item jid='osad-6993a948ba at spacewalk' subscription='both' /><item jid='osad-ff2a8b73b3 at spacewalk' subscription='both' /><item jid='osad-293b707684 at spacewalk' subscription='both' /><item jid='osad-c62f89446c at spacewalk' subscription='both' /><item jid='osad-77eecb67cc at spacewalk' subscription='both' /><item jid='osad-8af17b5e81 at spacewalk' subscription='both' /><item jid='osad-956277335c at spacewalk' subscription='both' /><item jid='osad-2c8fbbb064 at spacewalk' subscription='both' /><item jid='osad-22f5357ae2 at spacewalk' subscription='both' /><item jid='osad-86021fcab3 at spacewalk' subscription='both' /><item jid='osad-b6d3d1b223 at spacewalk' subscription='both' /><item jid='osad-2394e72059 at spacewalk' subscription='both' /><item jid='osad-825b3859fb at spacewalk' subscription='both' /><item jid='osad-ffd10a0231 at spacewalk' subscription='both' /><item jid='osad-ed088f4a2b at spacewalk' subscription='both' /><item jid='osad-1bebc55aaa at spacewalk' subscription='both' /><item jid='osad-5e3fa1057c at spacewalk' subscription='both' /></query></iq>)


Is it normal for this to happen?  We have about 120 - 130 servers in our environment so this doesn't nearly account for all of the clients (if it should account for all of them, that is).

As mentioned, server is 1.5 and is running on 64-bit CentOS, Postgres backend.

Any ideas on how to further troubleshoot?