<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>You should be able to upgrade to the new kombu package </div><div> </div><div><a href="http://koji.katello.org/koji/buildinfo?buildID=30314">http://koji.katello.org/koji/buildinfo?buildID=30314</a></div><div> </div><div>Thanks</div><div>Preethi</div><div> </div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"> <style></style> <div class="WordSection1"> Yes, thanks, that looks like it precisely. Is there a workaround, a new kombu package to upgrade to, or to regress to? -Alan <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Preethi Thomas [mailto:pthomas@redhat.com] Sent: Tuesday, April 05, 2016 10:52 AM To: Sparks, Alan <alan.l.sparks@hpe.com> Cc: pulp-list@redhat.com Subject: Re: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7 </div> </div> <div> <div> Looks like you may be hitting this issue </div> <div> </div> <div> <a href="https://pulp.plan.io/issues/1801" target="_blank">https://pulp.plan.io/issues/1801</a> </div> <div> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="100%" align="center" id="zwchr"> </div> <blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt"> The only thing I’ve found in the syslogs, before all goes south is: Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-8@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone missing, removing from list of workers Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-8@oa-ftc-repo0001.oa.ftc.hpelabs.net" target="_blank">reserved_resource_worker-8@oa-ftc-repo0001.oa.ftc.hpelabs.net</a> is missing. Canceling the tasks in its queue. Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-6@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone missing, removing from list of workers Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-6@oa-ftc-repo0001.oa.ftc.hpelabs.net" target="_blank">reserved_resource_worker-6@oa-ftc-repo0001.oa.ftc.hpelabs.net</a> is missing. Canceling the tasks in its queue. Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-4@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone missing, removing from list of workers Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-4@oa-ftc-repo0001.oa.ftc.hpelabs.net" target="_blank">reserved_resource_worker-4@oa-ftc-repo0001.oa.ftc.hpelabs.net</a> is missing. Canceling the tasks in its queue. Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-5@oa-ftc-repo0001.oa.ftc.hpelabs.net' has gone missing, removing from list of workers No idea what is happening at that time, that would cause all the processes to “disconnect”. They seem to still be running… qpid or mongo going bad somehow? Just vanilla stock installs of those. -Alan <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: <a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a> [<a href="mailto:pulp-list-bounces@redhat.com" target="_blank">mailto:pulp-list-bounces@redhat.com</a>] On Behalf Of Sparks, Alan Sent: Tuesday, April 05, 2016 10:19 AM To: <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7 </div> </div> I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running all from the pulp-stable distribution. QPID as MQ, and Mongo server mongodb-server-2.4.14-1.el6.x86_64. I can’t seem to keep it running more than a week before it falls over, tasks stop running, and following repeated in the syslog: Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running. Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running. I do see this running in process list: 10606 ? Sl 7:57 /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/ -f /var/log/pulp/celerybeat.log -l INFO --detach --pidfile=/var/run/pulp/celerybeat.pid If I attempt to stop celerybeat: # service pulp_celerybeat stop celery init v10.0. Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat Stopping pulp_celerybeat... ERROR Timed out while stopping (30s) I’m not sure how to determine what it dying… If I hard stop and start everything, or if I reboot, I can get out of the issue for a few days, before it recurs. Does anyone have advice on what to look for? The Pulp logs basically says everything is logging to syslog, but have not found a smoking gun to indicate what fell over. -Alan _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </blockquote> <div> </div> </div> </div> </blockquote><div> </div></div></body></html>