[Pulp-list] Pulp 2.7.1 with RabbitMQ

Tue Dec 22 17:37:13 UTC 2015

I'm trying to setup Pulp 2.7.1 with RabbitMQ and I was able to create a new repo for test but I cannot delete anymore as I get the following error:
[root at lab-a-pulp001 log]# pulp-admin -u admin -p admin rpm repo delete --repo-id=ol6_latest_20150707An internal error occurred on the Pulp server:
RequestException: DELETE requeston /pulp/api/v2/repositories/ol6_latest_20150707/ failed with 500 - client:0-10, server: 9-1
In /var/log/messages I have:Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled ExceptionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408) client: 0-10, server: 9-1Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408) Traceback (most recent call last):Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 109, in get_responseDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     response = callback(request, *callback_args, **callback_kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 48, in viewDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     return self.dispatch(request, *args, **kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 69, in dispatchDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     return handler(request, *args, **kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/views/decorators.py", line 237, in _auth_decoratorDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/views/decorators.py", line 191, in _verify_authDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     value = method(self, *args, **kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/views/repositories.py", line 286, in deleteDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     [repo_id], tags=task_tags)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 323, in apply_async_with_reservationDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     queue=RESOURCE_MANAGER_QUEUE)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 555, in apply_asyncDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     **dict(self._get_exec_options(), **options)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/celery/app/base.py", line 351, in send_taskDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     reply_to=reply_to or self.oid, **optionsDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/celery/app/amqp.py", line 305, in publish_taskDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     **kwargsDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 168, in publishDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     routing_key, mandatory, immediate, exchange, declare)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 457, in _ensuredDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     interval_max)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 369, in ensure_connectionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     interval_start, interval_step, interval_max, callback)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/utils/__init__.py", line 243, in retry_over_timeDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     return fun(*args, **kwargs)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 237, in connectDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     return self.connectionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 741, in connectionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     self._connection = self._establish_connection()Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 696, in _establish_connectionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     conn = self.transport.establish_connection()Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1700, in establish_connectionDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     conn = self.Connection(**opts)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1308, in __init__Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     self._qpid_conn = establish(**self.connection_options)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 68, in establishDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     conn.open(timeout=timeout)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "<string>", line 6, in openDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 268, in openDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     self.attach(timeout=timeout)Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "<string>", line 6, in attachDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 286, in attachDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 219, in _ewaitDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     self.check_error()Dec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in check_errorDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408)     raise eDec 22 17:31:09 lab-a-pulp001 pulp: pulp.server.webservices.middleware.exception:ERROR: (1611-13408) VersionError: client: 0-10, server: 9-1Dec 22 17:31:13 lab-a-pulp001 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.Dec 22 17:31:13 lab-a-pulp001 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_celerybeat processes running. Pulp will not operate correctly without at least one pulp_celerybeat process running.Dec 22 17:31:13 lab-a-pulp001 pulp: pulp.server.async.scheduler:ERROR: client: 0-10, server: 9-1Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376) Trying again in 12.00 seconds...Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376)Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (2877-59936) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (2877-59936) Trying again in 18.00 seconds...Dec 22 17:31:13 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (2877-59936)Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3091-54208) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3091-54208) Trying again in 12.00 seconds...Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3091-54208)Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3118-74624) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3118-74624) Trying again in 12.00 seconds...Dec 22 17:31:14 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3118-74624)Dec 22 17:31:15 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3145-16960) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:15 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3145-16960) Trying again in 12.00 seconds...Dec 22 17:31:15 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3145-16960)Dec 22 17:31:23 lab-a-pulp001 pulp: pulp.server.async.scheduler:ERROR: client: 0-10, server: 9-1Dec 22 17:31:25 lab-a-pulp001 pulp: celery.beat:ERROR: beat: Connection error: client: 0-10, server: 9-1. Trying again in 24.0 seconds...Dec 22 17:31:25 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376) consumer: Cannot connect to qpid://localhost:5672//: client: 0-10, server: 9-1.Dec 22 17:31:25 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376) Trying again in 14.00 seconds...Dec 22 17:31:25 lab-a-pulp001 pulp: celery.worker.consumer:ERROR: (3066-17376)

Nearly no special configuration in /etc/pulp/server.conf apart from setting the transport to rabbitmq:[root at lab-a-pulp001 log]# egrep -v '^($|#)' /etc/pulp/server.conf[database][server][authentication][security][consumer_history][data_reaping][oauth][messaging]transport: rabbitmq[tasks][email]
My RPM list:[root at lab-a-pulp001 log]# rpm -qa | egrep 'pulp|qpid' | sortmod_wsgi-3.4-2.pulp.el6.x86_64pulp-admin-client-2.7.1-1.el6.noarchpulp-puppet-admin-extensions-2.7.1-1.el6.noarchpulp-puppet-plugins-2.7.1-1.el6.noarchpulp-rpm-admin-extensions-2.7.1-1.el6.noarchpulp-rpm-plugins-2.7.1-1.el6.noarchpulp-selinux-2.7.1-1.el6.noarchpulp-server-2.7.1-1.el6.noarchpython-isodate-0.5.0-4.pulp.el6.noarchpython-kombu-3.0.24-10.pulp.el6.noarchpython-pulp-bindings-2.7.1-1.el6.noarchpython-pulp-client-lib-2.7.1-1.el6.noarchpython-pulp-common-2.7.1-1.el6.noarchpython-pulp-oid_validation-2.7.1-1.el6.noarchpython-pulp-puppet-common-2.7.1-1.el6.noarchpython-pulp-repoauth-2.7.1-1.el6.noarchpython-pulp-rpm-common-2.7.1-1.el6.noarchpython-qpid-0.26-3.el6.noarchpython-qpid-common-0.26-3.el6.noarchpython-qpid-qmf-0.26-2.el6.x86_64python-rhsm-1.8.0-2.pulp.el6.x86_64qpid-cpp-client-0.26-9.el6.x86_64qpid-proton-c-0.9-3.el6.x86_64qpid-qmf-0.26-2.el6.x86_64[root at lab-a-pulp001 log]#

Any pointers that I can follow?
Thank you very much,Radu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20151222/9d27f764/attachment.htm>