<div dir="ltr"><div>All sync jobs are now failing after upgrading from 2.6.4-1.  </div><div><br></div><div>Nov 13 18:43:11 Updated: python-pulp-common-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:12 Updated: python-pulp-puppet-common-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:12 Updated: python-pulp-rpm-common-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:12 Updated: python-gofer-2.6.6-2.el6.noarch</div><div>Nov 13 18:43:12 Updated: python-nectar-1.3.3-1.el6.noarch</div><div>Nov 13 18:43:13 Updated: python-pulp-bindings-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:13 Updated: python-pulp-client-lib-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:14 Updated: pulp-admin-client-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:14 Installed: python-pulp-repoauth-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:14 Installed: python-pulp-oid_validation-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:19 Installed: Django14-1.4.21-1.el6.noarch</div><div>Nov 13 18:43:19 Updated: python-bson-2.7.2-1.el6.x86_64</div><div>Nov 13 18:43:19 Updated: python-pymongo-2.7.2-1.el6.x86_64</div><div>Nov 13 18:43:20 Updated: python-pymongo-gridfs-2.7.2-1.el6.x86_64</div><div>Nov 13 18:43:20 Updated: python-mongoengine-0.9.0-1.el6.noarch</div><div>Nov 13 18:43:33 Updated: pulp-server-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:34 Updated: pulp-puppet-plugins-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:34 Updated: pulp-rpm-plugins-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:35 Updated: pulp-puppet-admin-extensions-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:35 Updated: pulp-rpm-admin-extensions-2.7.0-1.el6.noarch</div><div>Nov 13 18:43:35 Updated: python-gofer-qpid-2.6.6-2.el6.noarch</div><div><br></div><div><br></div><div>Nov 13 18:44:53 pulpplv01 pulp: pulp.server.db.manage:INFO: Beginning database migrations.</div><div>Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Applying pulp.server.db.migrations version 13</div><div>Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Migration to pulp.server.db.migrations version 13 complete.</div><div>Nov 13 18:44:54 pulpplv01 pulp: pulp.server.db.manage:INFO: Applying pulp.server.db.migrations version 14</div><div>Nov 13 18:44:54 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New worker '<a href="mailto:reserved_resource_worker-0@pulpplv01.domain.com">reserved_resource_worker-0@pulpplv01.domain.com</a>' discovered</div><div>Nov 13 18:44:55 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New worker '<a href="mailto:reserved_resource_worker-1@pulpplv01.domain.com">reserved_resource_worker-1@pulpplv01.domain.com</a>' discovered</div><div>Nov 13 18:44:56 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New worker '<a href="mailto:reserved_resource_worker-2@pulpplv01.domain.com">reserved_resource_worker-2@pulpplv01.domain.com</a>' discovered</div><div>Nov 13 18:44:57 pulpplv01 pulp: pulp.server.async.worker_watcher:INFO: New worker '<a href="mailto:reserved_resource_worker-3@pulpplv01.domain.com">reserved_resource_worker-3@pulpplv01.domain.com</a>' discovered</div><div>Nov 13 18:45:49 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '<a href="mailto:resource_manager@pulpplv01.domain.com">resource_manager@pulpplv01.domain.com</a>' has gone missing, removing from list of workers</div><div>Nov 13 18:45:49 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:resource_manager@pulpplv01.domain.com">resource_manager@pulpplv01.domain.com</a> is missing. Canceling the tasks in its queue.</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '<a href="mailto:reserved_resource_worker-0@pulpplv01.domain.com">reserved_resource_worker-0@pulpplv01.domain.com</a>' has gone missing, removing from list of workers</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-0@pulpplv01.domain.com">reserved_resource_worker-0@pulpplv01.domain.com</a> is missing. Canceling the tasks in its queue.</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '<a href="mailto:reserved_resource_worker-1@pulpplv01.domain.com">reserved_resource_worker-1@pulpplv01.domain.com</a>' has gone missing, removing from list of workers</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-1@pulpplv01.domain.com">reserved_resource_worker-1@pulpplv01.domain.com</a> is missing. Canceling the tasks in its queue.</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '<a href="mailto:reserved_resource_worker-2@pulpplv01.domain.com">reserved_resource_worker-2@pulpplv01.domain.com</a>' has gone missing, removing from list of workers</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-2@pulpplv01.domain.com">reserved_resource_worker-2@pulpplv01.domain.com</a> is missing. Canceling the tasks in its queue.</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: Worker '<a href="mailto:reserved_resource_worker-3@pulpplv01.domain.com">reserved_resource_worker-3@pulpplv01.domain.com</a>' has gone missing, removing from list of workers</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.tasks:ERROR: The worker named <a href="mailto:reserved_resource_worker-3@pulpplv01.domain.com">reserved_resource_worker-3@pulpplv01.domain.com</a> is missing. Canceling the tasks in its queue.</div><div>Nov 13 18:45:50 pulpplv01 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource</div><div>_mananger process running.</div><div><br></div><div><br></div><div>Installed pkgs:</div><div>python-pulp-rpm-common-2.7.0-1.el6.noarch</div><div>python-pulp-client-lib-2.7.0-1.el6.noarch</div><div>pulp-puppet-admin-extensions-2.7.0-1.el6.noarch</div><div>python-kombu-3.0.24-10.pulp.el6.noarch</div><div>python-isodate-0.5.0-4.pulp.el6.noarch</div><div>pulp-admin-client-2.7.0-1.el6.noarch</div><div>pulp-server-2.7.0-1.el6.noarch</div><div>pulp-rpm-admin-extensions-2.7.0-1.el6.noarch</div><div>python-pulp-common-2.7.0-1.el6.noarch</div><div>python-pulp-repoauth-2.7.0-1.el6.noarch</div><div>pulp-puppet-plugins-2.7.0-1.el6.noarch</div><div>python-rhsm-1.8.0-2.pulp.el6.x86_64</div><div>mod_wsgi-3.4-2.pulp.el6.x86_64</div><div>python-pulp-puppet-common-2.7.0-1.el6.noarch</div><div>python-pulp-bindings-2.7.0-1.el6.noarch</div><div>python-pulp-oid_validation-2.7.0-1.el6.noarch</div><div>pulp-rpm-plugins-2.7.0-1.el6.noarch</div><div>pulp-selinux-2.7.0-1.el6.noarch</div><div><br></div><div>I have reproduced the error on a clean install.</div><div><br></div><div>Any help would be greatly appreciated,</div><div><br></div><div>Joel</div></div>