[Spacewalk-list] Spacewalk 2.2: Clients are not picking up scheduled tasks

Tue Jun 16 12:54:27 UTC 2015

Just saw, that my answer did not get to the list...

Did you try to run rhn_check with -vv or run it through the debugger with "python -i -m pdb $(which rhn_check)"?

Robert

Am 15.06.2015 18:56 schrieb Brian Musson <mrbrian at gmail.com>:
>
> Hi I am trying to get to the bottom of this issue but I have hit a wall. Any hints would be greatly appreciated. The action gets scheduled but never gets picked up. From last week the jobs are still "Queued" and have not been picked up yet. I have a task to do this same patching to a larger pool of systems (almost 800) this week. Any direction or hints would be greatly appreciated.
>
>
> This action will be executed after 6/11/15 6:00:00 AM PDT
> This action's status is: Queued.
> This action has not yet been picked up.
>
> BM
>
> On Fri, Jun 12, 2015 at 1:09 PM, Brian Musson <mrbrian at gmail.com> wrote:
>>
>> Sorry to double post, but i thought it may be useful to show the output of the spacewalk proxy's /var/log/rhn/rhn_proxy_broker.log at the time of the run.
>>
>> 10.12.82.141 = client
>>
>> spacewalk-proxy# tail -f /var/log/rhn/rhn_proxy_broker.log
>>
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.41: proxy/apacheServer.__call__('New request, component proxy.broker',)
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: broker/rhnBroker.handler
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._serverCommo
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: broker/rhnBroker.__handleAction
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._clientCommo
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._forwardServer2Client
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/apacheHandler.cleanupHandler
>> 2015/06/12 16:06:26 -04:00 11781 192.168.1.208: proxy/apacheServer.__call__('New request, component proxy.broker',)
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: broker/rhnBroker.handler
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._serverCommo
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: broker/rhnBroker.__handleAction
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._clientCommo
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._forwardServer2Client
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/apacheHandler.cleanupHandler
>> 2015/06/12 16:06:26 -04:00 11777 10.12.72.56: proxy/apacheServer.__call__('New request, component proxy.broker',)
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: broker/rhnBroker.handler
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._serverCommo
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: broker/rhnBroker.__handleAction
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._clientCommo
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._forwardServer2Client
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/apacheHandler.cleanupHandler
>> 2015/06/12 16:06:26 -04:00 11784 10.12.72.41: proxy/apacheServer.__call__('New request, component proxy.broker',)
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: broker/rhnBroker.handler
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._serverCommo
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: broker/rhnBroker.__handleAction
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._clientCommo
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._forwardServer2Client
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/apacheHandler.cleanupHandler
>>
>> BM
>>
>> On Fri, Jun 12, 2015 at 12:37 PM, Brian Musson <mrbrian at gmail.com> wrote:
>>>
>>> My server connects indirectly through a proxy due to network segmentation. Checked the proxy for that file but could not find it. I did a tail on the spacewalk master and saw lots of messages mentioning the proxy server that serves the clients in question:
>>>
>>> 10.12.13.142 = our proxy
>>> 1000010038 = client ID
>>>
>>> spacewalk-master# tail -f /var/log/rhn/rhn_server_xmlrpc.log | grep 10.12.13.142
>>> 2015/06/12 12:34:14 -07:00 16704 10.12.13.142: xmlrpc/queue.get(1000010038, 2, 'checkins enabled')
>>> 2015/06/12 12:34:15 -07:00 16718 10.12.13.142: xmlrpc/up2date.listChannels(1000010038,)
>>> 2015/06/12 12:34:15 -07:00 16721 10.12.13.142: xmlrpc/registration.welcome_message('lang: None',)
>>>
>>>
>>>
>>> BM
>>>
>>> On Fri, Jun 12, 2015 at 5:04 AM, Jan Dobes <jdobes at redhat.com> wrote:
>>>>
>>>> ----- Original Message -----
>>>> > From: "Brian Musson" <mrbrian at gmail.com>
>>>> > To: spacewalk-list at redhat.com
>>>> > Sent: Thursday, June 11, 2015 8:43:47 PM
>>>> > Subject: [Spacewalk-list] Spacewalk 2.2: Clients are not picking up   scheduled tasks
>>>> >
>>>> > I have about 3000 systems registered in spacewalk, but today we are focusing
>>>> > on applying package updates to 22 of them. Of the 22 systems scheduled to
>>>> > have security errata applied to them, 20 successfully completed the update
>>>> > without error. Unfortunately, there are two systems which have the task
>>>> > queued and have not picked it up yet.
>>>> >
>>>> > I have restarted osad, rhnsd, restarted jabberd on the spacewalk master and
>>>> > proxy through which these failed systems connect. Other clients which have
>>>> > successfully updated go through this proxy server as well.
>>>> >
>>>> > When looking at the GUI, the client appears to be healthy.
>>>> >
>>>> > BM
>>>> >
>>>>
>>>> What will appear in '/var/log/rhn/rhn_server_xmlrpc.log' on the spacewalk server when you run rhn_check?
>>>>
>>>> --
>>>> Jan Dobes
>>>> Satellite Engineering, Red Hat
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>>
>>
>