<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 12 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> Nevermind, things are looking prettier now. I unregistered that consumer again, deleted a ton of tasks that were in Waiting state, and rebooted the pulp server for good measure. I restarted goferd on the consumer and reregistered it. Seems fine now. It was like that particular consumer was filling up the qpid message queue or something along those lines. Also, before I rebooted the pulp server, I tried restarting pulp_workers again, but it was spinning its wheels/waiting trying to shut down one of the workers. <o:p></o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: pulp-list-bounces@redhat.com [mailto:pulp-list-bounces@redhat.com] On Behalf Of Ashby, Jason (IMS) Sent: Thursday, November 13, 2014 4:13 PM To: pulp-list@redhat.com Subject: [Pulp-list] ConnectionError: Enqueue capacity threshold exceeded on queue<o:p></o:p> </div> </div> <o:p> </o:p> Consumer and server: Pulp 2.4.3, CentOS 6.5. The consumers were upgraded in-place from 2.3. The server is a fresh 2.4.3 install.<o:p></o:p> <o:p> </o:p> Hi guys,<o:p></o:p> I keep hitting this error that seems to be clogging up my pulp server and preventing consumers from communicating with the server. This ERROR seems to complain about the same consumer (log08.example.com), so I’ve tried unregistering and reregistering that consumer from both the pulp server and consumer sides, as well as yum reinstalling pulp and gofer, but no luck. <o:p></o:p> <o:p> </o:p> I see some <a href="https://www.redhat.com/archives/pulp-list/2011-March/msg00092.html"> old discussions</a> about this, but no clear solution. Sounds like an bug that was resolved in newer versions of qpidd (>0.24-ish) Any ideas, or troubleshooting tips? <o:p></o:p> <o:p> </o:p> Jason<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Here is most of the traceback from /var/log/messages on the server:<o:p></o:p> <o:p> </o:p> …snip…<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: value = method(self, *args, **kwargs)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/consumers.py", line 238, in POST<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: consumer_id, repo_id, distributor_id, notify_agent, binding_config, options)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/tasks/consumer.py", line 52, in bind<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: task = agent_manager.bind(consumer_id, repo_id, distributor_id, agent_options)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/consumer/agent.py", line 98, in bind<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: agent.consumer.bind(context, agent_bindings, options)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/agent/direct/pulpagent.py", line 140, in bind<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: consumer.bind(bindings, options)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 72, in __call__<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: return self.stub._send(request, opts)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 142, in _send<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: return self.__send(request, options)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 173, in __send<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: any=opts.any)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 354, in send<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: trigger()<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 468, in __call__<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: self.__send()<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 450, in __send<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: queue.declare(policy.url)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/gofer/transport/qpid/model.py", line 156, in declare<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: sender = session.sender(self.address())<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "<string>", line 6, in sender<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 606, in sender<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: sender._ewait(lambda: sender.linked)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 814, in _ewait<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.session._ewait(lambda: self.error or predicate(), timeout)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 580, in _ewait<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.connection._ewait(lambda: self.error or predicate(), timeout)<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: self.check_error()<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in check_error<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: raise e<o:p></o:p> Nov 13 15:11:45 pulp-srv-01 pulp: pulp.server.webservices.middleware.exception:ERROR: ConnectionError: Enqueue capacity threshold exceeded on queue "pulp.agent.log08.example.com". (/builddir/build/BUILD/qpid-0.26/cpp/src/qpid/legacystore/JournalImpl.cpp:594)(501)<o:p></o:p> <o:p> </o:p> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="100%" align="center"> </div> Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.<o:p></o:p> </div> <hr> Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. </body> </html>