[Spacewalk-list] Spacewalk 2.2: Clients are not picking up scheduled tasks
Brian Musson
mrbrian at gmail.com
Tue Jun 16 14:59:05 UTC 2015
Robert, here is the output from the client:
# service osad status
osad (pid 1842) is running...
# rhn-actions-control --report
deploy is enabled
diff is enabled
upload is enabled
mtime_upload is enabled
run is enabled
# rhn_check -vvv
D: opening db environment /var/lib/rpm cdb:mpool:joinenv
D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
D: locked db index /var/lib/rpm/Packages
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening db index /var/lib/rpm/Name rdonly mode=0x0
D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
D: added key gpg-pubkey-0608b895-4bd22942 to keyring
D: added key gpg-pubkey-442df0f8-4783f24a to keyring
D: added key gpg-pubkey-548c16bf-4c29a642 to keyring
D: added key gpg-pubkey-217521f6-45e8a532 to keyring
D: added key gpg-pubkey-863a853d-4f55f54d to keyring
D: added key gpg-pubkey-1bb35891-54f75af8 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: opening db index /var/lib/rpm/Providename rdonly mode=0x0
D: do_call packages.checkNeedUpdate('rhnsd=1',){}
D: opening db environment /var/lib/rpm cdb:mpool:joinenv
D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening db index /var/lib/rpm/Name rdonly mode=0x0
D: added key gpg-pubkey-c105b9de-4e0fd3a3 to keyring
D: added key gpg-pubkey-0608b895-4bd22942 to keyring
D: added key gpg-pubkey-442df0f8-4783f24a to keyring
D: added key gpg-pubkey-548c16bf-4c29a642 to keyring
D: added key gpg-pubkey-217521f6-45e8a532 to keyring
D: added key gpg-pubkey-863a853d-4f55f54d to keyring
D: added key gpg-pubkey-1bb35891-54f75af8 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: opening db index /var/lib/rpm/Providename rdonly mode=0x0
D: closed db index /var/lib/rpm/Providename
D: closed db index /var/lib/rpm/Name
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
Loaded plugins: fastestmirror, presto, rhnplugin
Config time: 0.038
D: login(forceUpdate=False) invoked
D: readCachedLogin invoked
D: Unable to read pickled loginInfo at: /var/spool/up2date/loginAuth.pkl
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1434032930.09 with expiration of 1434036530.09 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010314, 'X-RHN-Auth-Server-Time': '1434032926.38', 'X-RHN-Auth': '7RfB/XtV6EqZw8hGYqe+dFasQ+3q9QvfIzO+RrKIdd0=', 'X-RHN-Auth-Channels': [['centos6-base-x86_64', '20150611060001', '1', '1'], ['spacewalk-client-x86_64', '20150611061143', '0', '1'], ['epel6-x86_64', '20150611061143', '0', '1'], ['centos6-sysops-x86_64', '20150504165024', '0', '1'], ['centos6-updates-x86_64', '20150611060258', '0', '1']], 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: rpcServer: Calling XMLRPC up2date.listChannels
This system is receiving updates from RHN Classic or Red Hat Satellite.
Setting up Package Sacks
Loading mirror speeds from cached hostfile
pkgsack time: 0.065
rpmdb time: 0.000
Loading mirror speeds from cached hostfile
repo time: 0.000
D: local action status: (0, 'rpm database not modified since last update (or package list recently updated)', {})
D: rpcServer: Calling XMLRPC registration.welcome_message
D: closed db index /var/lib/rpm/Providename
D: closed db index /var/lib/rpm/Name
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
> On Jun 16, 2015, at 05:54, Robert Paschedag <robert.paschedag at web.de> wrote:
>
> Just saw, that my answer did not get to the list...
>
> Did you try to run rhn_check with -vv or run it through the debugger with "python -i -m pdb $(which rhn_check)"?
>
> Robert
>
> Am 15.06.2015 18:56 schrieb Brian Musson <mrbrian at gmail.com>:
>>
>> Hi I am trying to get to the bottom of this issue but I have hit a wall. Any hints would be greatly appreciated. The action gets scheduled but never gets picked up. From last week the jobs are still "Queued" and have not been picked up yet. I have a task to do this same patching to a larger pool of systems (almost 800) this week. Any direction or hints would be greatly appreciated.
>>
>>
>> This action will be executed after 6/11/15 6:00:00 AM PDT
>> This action's status is: Queued.
>> This action has not yet been picked up.
>>
>> BM
>>
>>> On Fri, Jun 12, 2015 at 1:09 PM, Brian Musson <mrbrian at gmail.com> wrote:
>>>
>>> Sorry to double post, but i thought it may be useful to show the output of the spacewalk proxy's /var/log/rhn/rhn_proxy_broker.log at the time of the run.
>>>
>>> 10.12.82.141 = client
>>>
>>> spacewalk-proxy# tail -f /var/log/rhn/rhn_proxy_broker.log
>>>
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.41: proxy/apacheServer.__call__('New request, component proxy.broker',)
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: broker/rhnBroker.handler
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._serverCommo
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: broker/rhnBroker.__handleAction
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._clientCommo
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/rhnShared._forwardServer2Client
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>>> 2015/06/12 16:06:26 -04:00 11778 10.12.82.141: proxy/apacheHandler.cleanupHandler
>>> 2015/06/12 16:06:26 -04:00 11781 192.168.1.208: proxy/apacheServer.__call__('New request, component proxy.broker',)
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: broker/rhnBroker.handler
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._serverCommo
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: broker/rhnBroker.__handleAction
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._clientCommo
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/rhnShared._forwardServer2Client
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>>> 2015/06/12 16:06:26 -04:00 11781 10.12.82.141: proxy/apacheHandler.cleanupHandler
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.72.56: proxy/apacheServer.__call__('New request, component proxy.broker',)
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: broker/rhnBroker.handler
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._serverCommo
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: broker/rhnBroker.__handleAction
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._clientCommo
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/rhnShared._forwardServer2Client
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>>> 2015/06/12 16:06:26 -04:00 11777 10.12.82.141: proxy/apacheHandler.cleanupHandler
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.72.41: proxy/apacheServer.__call__('New request, component proxy.broker',)
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: broker/rhnBroker.handler
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._serverCommo
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: broker/rhnBroker.__handleAction
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._clientCommo
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/rhnShared._forwardServer2Client
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/apacheHandler.handler('Leaving with status code 0',)
>>> 2015/06/12 16:06:26 -04:00 11784 10.12.82.141: proxy/apacheHandler.cleanupHandler
>>>
>>> BM
>>>
>>>> On Fri, Jun 12, 2015 at 12:37 PM, Brian Musson <mrbrian at gmail.com> wrote:
>>>>
>>>> My server connects indirectly through a proxy due to network segmentation. Checked the proxy for that file but could not find it. I did a tail on the spacewalk master and saw lots of messages mentioning the proxy server that serves the clients in question:
>>>>
>>>> 10.12.13.142 = our proxy
>>>> 1000010038 = client ID
>>>>
>>>> spacewalk-master# tail -f /var/log/rhn/rhn_server_xmlrpc.log | grep 10.12.13.142
>>>> 2015/06/12 12:34:14 -07:00 16704 10.12.13.142: xmlrpc/queue.get(1000010038, 2, 'checkins enabled')
>>>> 2015/06/12 12:34:15 -07:00 16718 10.12.13.142: xmlrpc/up2date.listChannels(1000010038,)
>>>> 2015/06/12 12:34:15 -07:00 16721 10.12.13.142: xmlrpc/registration.welcome_message('lang: None',)
>>>>
>>>>
>>>>
>>>> BM
>>>>
>>>>> On Fri, Jun 12, 2015 at 5:04 AM, Jan Dobes <jdobes at redhat.com> wrote:
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Brian Musson" <mrbrian at gmail.com>
>>>>>> To: spacewalk-list at redhat.com
>>>>>> Sent: Thursday, June 11, 2015 8:43:47 PM
>>>>>> Subject: [Spacewalk-list] Spacewalk 2.2: Clients are not picking up scheduled tasks
>>>>>>
>>>>>> I have about 3000 systems registered in spacewalk, but today we are focusing
>>>>>> on applying package updates to 22 of them. Of the 22 systems scheduled to
>>>>>> have security errata applied to them, 20 successfully completed the update
>>>>>> without error. Unfortunately, there are two systems which have the task
>>>>>> queued and have not picked it up yet.
>>>>>>
>>>>>> I have restarted osad, rhnsd, restarted jabberd on the spacewalk master and
>>>>>> proxy through which these failed systems connect. Other clients which have
>>>>>> successfully updated go through this proxy server as well.
>>>>>>
>>>>>> When looking at the GUI, the client appears to be healthy.
>>>>>>
>>>>>> BM
>>>>>
>>>>> What will appear in '/var/log/rhn/rhn_server_xmlrpc.log' on the spacewalk server when you run rhn_check?
>>>>>
>>>>> --
>>>>> Jan Dobes
>>>>> Satellite Engineering, Red Hat
>>>>>
>>>>> _______________________________________________
>>>>> Spacewalk-list mailing list
>>>>> Spacewalk-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20150616/ddbbc748/attachment.htm>
More information about the Spacewalk-list
mailing list