<div dir="ltr">I submitted this a couple day ago. Subject: Workers disappearing / canceled tasks after upgrading to 2.7.0.1 - CentOS 6.7<div> </div><div>Can these be merged? <div class="gmail_extra"> <div class="gmail_quote">On Wed, Nov 18, 2015 at 9:05 AM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Jason and Jeffrey, Thanks for reporting this. I've written up a bug [0] and I am investigating the root cause. On the bug are you able to leave some answers to these questions? - Can you confirm that it affects both RabbitMQ and Qpid usage? - Can you confirm that the workers "go missing" and then return, and then "go missing" in a continuous cycle? I expect it to happen every 90 seconds. - Jeffrey specifically, what OS are you using? [0]: <a href="https://pulp.plan.io/issues/1380" rel="noreferrer" target="_blank">https://pulp.plan.io/issues/1380</a> Thanks, Brian On 11/18/2015 09:33 AM, Miller, Jeffrey L wrote: > I am seeing this behavior as well after upgrading from 2.6 to 2.7. > However, I am using qpid not rabbitmq. > > > > -Jeffrey > > > > > > > > *From:* <a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a> > [mailto:<a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a>] *On Behalf Of *Ashby, Jason (IMS) > *Sent:* Wednesday, November 18, 2015 8:29 AM > *To:* <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> > *Subject:* [Pulp-list] Missing celery workers <div><div>> > > > Hi all, > > I’m hitting another issue with the upgrade to Pulp 2.7.0 + changing from > qpid to rabbitmq for messaging. The workers are continuously going > missing, every minute or so. The effect is that the tasks in the task > list stay in a Waiting state and are never completed. > > > > Rabbitmq looks healthy; I see successful accepted connections per the > logs and can see a bunch of connections in the rabbitmq management GUI. > I’m kind of stuck as far as troubleshooting goes. Any tips on what else > to investigate? > > > > Pulp and rabbitmq servers are both CentOS 6. > > > > # /var/log/messages > > Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed > heartbeat from resource_manager@pulp01 > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New > worker 'reserved_resource_worker-3@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New > worker 'reserved_resource_worker-1@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New > worker 'reserved_resource_worker-2@pulp01' discovered > > Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New > worker 'reserved_resource_worker-0@pulp01' discovered > > Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New > worker 'resource_manager@pulp01' discovered > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-3@pulp01' has gone missing, removing from list > of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks > in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-1@pulp01' has gone missing, removing from list > of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks > in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-2@pulp01' has gone missing, removing from list > of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks > in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'reserved_resource_worker-0@pulp01' has gone missing, removing from list > of work > > ers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks > in its q > > ueue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker > 'resource_manager@pulp01' has gone missing, removing from list of workers > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker > named resource_manager@pulp01 is missing. Canceling the tasks in its queue. > > Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There > are 0 pulp_resource_manager processes running. Pulp will not operate > correctly without > > at least one pulp_resource_mananger process running. > > > </div></div>> ------------------------------------------------------------------------ > > > Information in this e-mail may be confidential. It is intended only for > the addressee(s) identified above. If you are not the addressee(s), or > an employee or agent of the addressee(s), please note that any > dissemination, distribution, or copying of this communication is > strictly prohibited. If you have received this e-mail in error, please > notify the sender of the error. > > > > _______________________________________________ > Pulp-list mailing list > <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> > _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </blockquote></div> </div></div></div>