[Pulp-list] CentOS 6.7 - Workers disappearing / canceled tasks after upgrading to 2.7.0.1

[Joel Golden]

All sync jobs are now failing after upgrading from 2.6.4-1.

Nov 13 18:43:11 Updated: python-pulp-common-2.7.0-1.el6.noarch
Nov 13 18:43:12 Updated: python-pulp-puppet-common-2.7.0-1.el6.noarch
Nov 13 18:43:12 Updated: python-pulp-rpm-common-2.7.0-1.el6.noarch
Nov 13 18:43:12 Updated: python-gofer-2.6.6-2.el6.noarch
Nov 13 18:43:12 Updated: python-nectar-1.3.3-1.el6.noarch
Nov 13 18:43:13 Updated: python-pulp-bindings-2.7.0-1.el6.noarch
Nov 13 18:43:13 Updated: python-pulp-client-lib-2.7.0-1.el6.noarch
Nov 13 18:43:14 Updated: pulp-admin-client-2.7.0-1.el6.noarch
Nov 13 18:43:14 Installed: python-pulp-repoauth-2.7.0-1.el6.noarch
Nov 13 18:43:14 Installed: python-pulp-oid_validation-2.7.0-1.el6.noarch
Nov 13 18:43:19 Installed: Django14-1.4.21-1.el6.noarch
Nov 13 18:43:19 Updated: python-bson-2.7.2-1.el6.x86_64
Nov 13 18:43:19 Updated: python-pymongo-2.7.2-1.el6.x86_64
Nov 13 18:43:20 Updated: python-pymongo-gridfs-2.7.2-1.el6.x86_64
Nov 13 18:43:20 Updated: python-mongoengine-0.9.0-1.el6.noarch
Nov 13 18:43:33 Updated: pulp-server-2.7.0-1.el6.noarch
Nov 13 18:43:34 Updated: pulp-puppet-plugins-2.7.0-1.el6.noarch
Nov 13 18:43:34 Updated: pulp-rpm-plugins-2.7.0-1.el6.noarch
Nov 13 18:43:35 Updated: pulp-puppet-admin-extensions-2.7.0-1.el6.noarch
Nov 13 18:43:35 Updated: pulp-rpm-admin-extensions-2.7.0-1.el6.noarch
Nov 13 18:43:35 Updated: python-gofer-qpid-2.6.6-2.el6.noarch


Nov 13 18:44:53 pulpplv01 pulp: pulp.server.db.manage:INFO: Beginning
database migrations.
Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Applying
pulp.server.db.migrations version 13
Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Migration to
pulp.server.db.migrations version 13 complete.
Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Applying
pulp.server.db.migrations version 14
Nov 13 18:44:54 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New
worker 'reserved_resource_worker-0 at pulpplv01.domain.com' discovered
Nov 13 18:44:55 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New
worker 'reserved_resource_worker-1 at pulpplv01.domain.com' discovered
Nov 13 18:44:56 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New
worker 'reserved_resource_worker-2 at pulpplv01.domain.com' discovered
Nov 13 18:44:57 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New
worker 'reserved_resource_worker-3 at pulpplv01.domain.com' discovered
Nov 13 18:45:49 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '
resource_manager at pulpplv01.domain.com' has gone missing, removing from list
of workers
Nov 13 18:45:49 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker
named resource_manager at pulpplv01.domain.com is missing. Canceling the tasks
in its queue.
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '
reserved_resource_worker-0 at pulpplv01.domain.com' has gone missing, removing
from list of workers
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker
named reserved_resource_worker-0 at pulpplv01.domain.com is missing. Canceling
the tasks in its queue.
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '
reserved_resource_worker-1 at pulpplv01.domain.com' has gone missing, removing
from list of workers
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker
named reserved_resource_worker-1 at pulpplv01.domain.com is missing. Canceling
the tasks in its queue.
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '
reserved_resource_worker-2 at pulpplv01.domain.com' has gone missing, removing
from list of workers
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker
named reserved_resource_worker-2 at pulpplv01.domain.com is missing. Canceling
the tasks in its queue.
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '
reserved_resource_worker-3 at pulpplv01.domain.com' has gone missing, removing
from list of workers
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker
named reserved_resource_worker-3 at pulpplv01.domain.com is missing. Canceling
the tasks in its queue.
Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: There
are 0 pulp_resource_manager processes running. Pulp will not operate
correctly without at least one pulp_resource
_mananger process running.


Installed pkgs:
python-pulp-rpm-common-2.7.0-1.el6.noarch
python-pulp-client-lib-2.7.0-1.el6.noarch
pulp-puppet-admin-extensions-2.7.0-1.el6.noarch
python-kombu-3.0.24-10.pulp.el6.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
pulp-admin-client-2.7.0-1.el6.noarch
pulp-server-2.7.0-1.el6.noarch
pulp-rpm-admin-extensions-2.7.0-1.el6.noarch
python-pulp-common-2.7.0-1.el6.noarch
python-pulp-repoauth-2.7.0-1.el6.noarch
pulp-puppet-plugins-2.7.0-1.el6.noarch
python-rhsm-1.8.0-2.pulp.el6.x86_64
mod_wsgi-3.4-2.pulp.el6.x86_64
python-pulp-puppet-common-2.7.0-1.el6.noarch
python-pulp-bindings-2.7.0-1.el6.noarch
python-pulp-oid_validation-2.7.0-1.el6.noarch
pulp-rpm-plugins-2.7.0-1.el6.noarch
pulp-selinux-2.7.0-1.el6.noarch

I have reproduced the error on a clean install.

Any help would be greatly appreciated,

Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20151115/e4ddd10a/attachment.htm>