[Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

[Preethi Thomas]

Looks like you may be hitting this issue 

https://pulp.plan.io/issues/1801 

----- Original Message -----

> The only thing I’ve found in the syslogs, before all goes south is:

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-8 at oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-8 at oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-6 at oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-6 at oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-4 at oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.tasks:ERROR: The
> worker named reserved_resource_worker-4 at oa-ftc-repo0001.oa.ftc.hpelabs.net
> is missing. Canceling the tasks in its queue.

> Apr 1 18:28:19 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR:
> Worker 'reserved_resource_worker-5 at oa-ftc-repo0001.oa.ftc.hpelabs.net' has
> gone missing, removing from list of workers

> No idea what is happening at that time, that would cause all the processes to
> “disconnect”. They seem to still be running… qpid or mongo going bad
> somehow? Just vanilla stock installs of those.

> -Alan

> From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com] On
> Behalf Of Sparks, Alan
> Sent: Tuesday, April 05, 2016 10:19 AM
> To: pulp-list at redhat.com
> Subject: [Pulp-list] Pulp 2.8.0 stops running on CentOS 6.7

> I have a 2.8.0 install running on a patched-up CentOS 6.7 machine, running
> all from the pulp-stable distribution. QPID as MQ, and Mongo server
> mongodb-server-2.4.14-1.el6.x86_64.

> I can’t seem to keep it running more than a week before it falls over, tasks
> stop running, and following repeated in the syslog:

> Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There
> are 0 pulp_resource_manager processes running. Pulp will not operate
> correctly without at least one pulp_resource_mananger process running.

> Apr 5 16:09:22 oa-ftc-repo0001 pulp: pulp.server.async.scheduler:ERROR: There
> are 0 pulp_celerybeat processes running. Pulp will not operate correctly
> without at least one pulp_celerybeat process running.

> I do see this running in process list:

> 10606 ? Sl 7:57 /usr/bin/python /usr/bin/celery beat
> --app=pulp.server.async.celery_instance.celery
> --scheduler=pulp.server.async.scheduler.Scheduler --workdir=/var/run/pulp/
> -f /var/log/pulp/celerybeat.log -l INFO --detach
> --pidfile=/var/run/pulp/celerybeat.pid

> If I attempt to stop celerybeat:

> # service pulp_celerybeat stop

> celery init v10.0.

> Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat

> Stopping pulp_celerybeat... ERROR

> Timed out while stopping (30s)

> I’m not sure how to determine what it dying… If I hard stop and start
> everything, or if I reboot, I can get out of the issue for a few days,
> before it recurs.

> Does anyone have advice on what to look for? The Pulp logs basically says
> everything is logging to syslog, but have not found a smoking gun to
> indicate what fell over.

> -Alan

> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20160405/139477df/attachment.htm>