[Pulp-list] Missing celery workers

Sun Nov 29 16:48:35 UTC 2015

That is correct.  Although behavior was different after the switch and
restart of services, (i.e. my run sync jobs did not immediately crash), I
still received the missing workers messages.  Unfortunately, the only fix I
found was to migrated to CentOS 7, which was much less painful than
debugging 6.

On Sun, Nov 29, 2015 at 9:43 AM, Ashby, Jason (IMS) <AshbyJ at imsweb.com>
wrote:

> Hmm, I don’t think this would fix the missing worker issue for people like
> myself using rabbitmq instead of qpid.  Though it may fix the* other*
> issue of broken EL6 qpid dependencies.
>
>
>
> *From:* pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com]
> *On Behalf Of *Joel Golden
> *Sent:* Thursday, November 26, 2015 12:27 AM
>
> *To:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] Missing celery workers
>
>
>
> Fix - replace qpid-cpp-server:
>
>
>
> Installing:
>
> qpid-cpp-server-linearstore x86_64 0.34-4.el6
>
> replacing qpid-cpp-server-store.x86_64 0.26-9.el6
>
> Updating:
>
> python-qpid noarch 0.32-12.el6
>
> python-qpid-common noarch 0.32-12.el6
>
> python-qpid-qmf x86_64 0.32-1.el6
>
> qpid-cpp-client x86_64 0.34-4.el6
>
> qpid-cpp-server x86_64 0.34-4.el6
>
> qpid-proton-c x86_64 0.10-2.el6
>
> qpid-qmf x86_64 0.32-1.el6
>
> Installing for dependencies:
>
> python-saslwrapper x86_64 0.22-5.el6
>
>
>
> Enjoy!  Joel Golden
>
>
>
> Due to an existing set of deprecated Qpid packages in RHEL 6 and CentOS 6,
> we cannot at present provide updated Qpid packages directly in EPEL 6. We
> offer an alternative repository at Fedora Copr.
>
> Qpid at Copr
>
> Copr repo file for RHEL 6 and CentOS 6
>
>
> https://copr.fedoraproject.org/coprs/irina/qpid/repo/epel-6/irina-qpid-epel-6.repo
>
> Copr Qpid GPG public key
>
> https://qpid.apache.org/copr-qpid-pubkey.gpg
>
>
>
> On Thu, Nov 19, 2015 at 5:08 AM, Miller, Jeffrey L <
> jeff-l-miller at uiowa.edu> wrote:
>
> In answer to posed questions:
> - RHEL 6 x86_64
> - qpid-cpp-server
> - Every 60 seconds for me as well. /var/log/messages loos as below.
>
> -Jeffrey
>
>
>
>
>
> -----Original Message-----
> From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com]
> On Behalf Of Ashby, Jason (IMS)
> Sent: Wednesday, November 18, 2015 10:38 AM
> To: Brian Bouterse <bbouters at redhat.com>; pulp-list at redhat.com
> Subject: Re: [Pulp-list] Missing celery workers
>
> For me it is consistently every 60 seconds. Here are logs from the last 10
> minutes.  I also set loglevel to DEBUG for each config file in
> /etc/default/pulp* and restarted the services, but I'm not seeing any DEBUG
> stuff in the logs.
>
>
> $ sudo grep missing /var/log/messages
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:23:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:24:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:25:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:25:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:26:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:27:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:28:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:29:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:30:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:31:34 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-0 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-2 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-1 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'reserved_resource_worker-3 at pulp01' has gone missing, removing from list
> of workers Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> tasks in its queue.
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> 'resource_manager at pulp01' has gone missing, removing from list of workers
> Nov 18 11:32:35 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
>
> -----Original Message-----
> From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com]
> On Behalf Of Brian Bouterse
> Sent: Wednesday, November 18, 2015 11:05 AM
> To: pulp-list at redhat.com
> Subject: Re: [Pulp-list] Missing celery workers
>
> Jason and Jeffrey,
>
> Thanks for reporting this. I've written up a bug [0] and I am
> investigating the root cause.
>
> On the bug are you able to leave some answers to these questions?
>
> - Can you confirm that it affects both RabbitMQ and Qpid usage?
> - Can you confirm that the workers "go missing" and then return, and then
> "go missing" in a continuous cycle? I expect it to happen every 90 seconds.
>
> - Jeffrey specifically, what OS are you using?
>
> [0]: https://pulp.plan.io/issues/1380
>
> Thanks,
> Brian
>
> On 11/18/2015 09:33 AM, Miller, Jeffrey L wrote:
> > I am seeing this behavior as well after upgrading from 2.6 to 2.7.
> > However, I am using qpid not rabbitmq.
> >
> >
> >
> > -Jeffrey
> >
> >
> >
> >
> >
> >
> >
> > *From:* pulp-list-bounces at redhat.com
> > [mailto:pulp-list-bounces at redhat.com] *On Behalf Of *Ashby, Jason
> > (IMS)
> > *Sent:* Wednesday, November 18, 2015 8:29 AM
> > *To:* pulp-list at redhat.com
> > *Subject:* [Pulp-list] Missing celery workers
> >
> >
> >
> > Hi all,
> >
> > I'm hitting another issue with the upgrade to Pulp 2.7.0 + changing
> > from qpid to rabbitmq for messaging.  The workers are continuously
> > going missing, every minute or so.  The effect is that the tasks in
> > the task list stay in a Waiting state and are never completed.
> >
> >
> >
> > Rabbitmq looks healthy; I see successful accepted connections per the
> > logs and can see a bunch of connections in the rabbitmq management GUI.
> > I'm kind of stuck as far as troubleshooting goes.  Any tips on what
> > else to investigate?
> >
> >
> >
> > Pulp and rabbitmq servers are both CentOS 6.
> >
> >
> >
> > # /var/log/messages
> >
> > Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed
> > heartbeat from resource_manager at pulp01
> >
> > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-3 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-1 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-2 at pulp01' discovered
> >
> > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO:
> > New worker 'reserved_resource_worker-0 at pulp01' discovered
> >
> > Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO:
> > New worker 'resource_manager at pulp01' discovered
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-3 at pulp01' has gone missing, removing from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> > named reserved_resource_worker-3 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-1 at pulp01' has gone missing, removing from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> > named reserved_resource_worker-1 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-2 at pulp01' has gone missing, removing from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> > named reserved_resource_worker-2 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> > 'reserved_resource_worker-0 at pulp01' has gone missing, removing from
> > list of work
> >
> > ers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> > named reserved_resource_worker-0 at pulp01 is missing. Canceling the
> > tasks in its q
> >
> > ueue.
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker
> > 'resource_manager at pulp01' has gone missing, removing from list of
> > workers
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker
> > named resource_manager at pulp01 is missing. Canceling the tasks in its
> queue.
> >
> > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There
> > are 0 pulp_resource_manager processes running. Pulp will not operate
> > correctly without
> >
> > at least one pulp_resource_mananger process running.
> >
> >
> >
> > ----------------------------------------------------------------------
> > --
> >
> >
> > Information in this e-mail may be confidential. It is intended only
> > for the addressee(s) identified above. If you are not the
> > addressee(s), or an employee or agent of the addressee(s), please note
> > that any dissemination, distribution, or copying of this communication
> > is strictly prohibited. If you have received this e-mail in error,
> > please notify the sender of the error.
> >
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> ________________________________
>
> Information in this e-mail may be confidential. It is intended only for
> the addressee(s) identified above. If you are not the addressee(s), or an
> employee or agent of the addressee(s), please note that any dissemination,
> distribution, or copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify the sender of the
> error.
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
> ------------------------------
>
> Information in this e-mail may be confidential. It is intended only for
> the addressee(s) identified above. If you are not the addressee(s), or an
> employee or agent of the addressee(s), please note that any dissemination,
> distribution, or copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify the sender of the
> error.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20151129/30341abc/attachment.htm>