[Pulp-list] Pulp 2.4: celery.beat:ERROR: Message Error: Couldn't apply scheduled task babysit: connection aborted

Tue Mar 25 09:38:15 UTC 2014

Hi John,

It seems that the connection to the qpid broker is not being made to allow Celery tasks to be dispatched.  The first thing I usually try it to run qpidd in the foreground with --auth=no as a command line option.  That way I know it's running, and that it knows that auth should be disabled.  The qpidd config file has moved some over the versions, so sometimes qpidd may not find the config file properly.  I run it temporarily in the foreground using:

sudo -u qpidd qpidd --auth=no

Can you reproduce the issue with qpidd running in the foreground?

-Brian

----- Original Message -----
From: "John Matthews" <jmatthew at redhat.com>
To: pulp-list at redhat.com
Sent: Monday, March 24, 2014 8:45:44 PM
Subject: [Pulp-list] Pulp 2.4: celery.beat:ERROR: Message Error: Couldn't apply scheduled task babysit: connection aborted

Hello,

I'm running pulp 2.4 (pulp-server-2.4.0-0.6.alpha.fc20.noarch) on Fedora 20 and running into the below error.  
Any thoughts on debugging this?

Mar 24 20:30:52 localhost pulp: celery.beat:ERROR: Message Error: Couldn't apply scheduled task babysit: connection aborted
['  File "/usr/bin/celery", line 9, in <module>\n    load_entry_point(\'celery==3.1.9\', \'console_scripts\', \'celery\')()\n', '  File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 30, in main\n    main()\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 80, in main\n    cmd.execute_from_commandline(argv)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 768, in execute_from_commandline\n    super(CeleryCommand, self).execute_from_commandline(argv)))\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 308, in execute_from_commandline\n    return self.handle_argv(self.prog_name, argv[1:])\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 760, in handle_argv\n    return self.execute(command, argv)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 692, in execute\n    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])\n', '  File "/usr/lib/p!
 ython2.7/site-packages/celery/bin/base.py", line 312, in run_from_argv\n    sys.argv if argv is None else argv, command)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 374, in handle_argv\n    return self(*args, **options)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 271, in __call__\n    ret = self.run(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/celery/bin/beat.py", line 79, in run\n    return beat().run()\n', '  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 82, in run\n    self.start_scheduler()\n', '  File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 111, in start_scheduler\n    beat.start()\n', '  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 444, in start\n    interval = self.scheduler.tick()\n', '  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 168, in tick\n    ret = super(Scheduler, self).tick()\n', '  File "/usr/lib!
 /python2.7/site-packages/celery/beat.py", line 213, in tick\n    next_
time_to_run = self.maybe_due(entry, self.publisher)\n', '  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 199, in maybe_due\n    exc, traceback.format_stack(), exc_info=True)\n']
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 196, in maybe_due
    result = self.apply_async(entry, publisher=publisher)
  File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 255, in apply_async
    result = super(Scheduler, self).apply_async(entry, publisher, **kwargs)
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 248, in apply_async
    entry, exc=exc)), sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/celery/beat.py", line 240, in apply_async
    **entry.options)
  File "/usr/lib/python2.7/site-packages/celery/app/task.py", line 555, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/usr/lib/python2.7/site-packages/celery/app/base.py", line 324, in send_task
    reply_to=reply_to or self.oid, **options
  File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 300, in publish_task
    **kwargs
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 168, in publish
    routing_key, mandatory, immediate, exchange, declare)
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 440, in _ensured
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 184, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 722, in basic_publish
    return self._put(routing_key, message, exchange, **kwargs)
  File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 497, in _put
    sender = self._qpid_session.sender(address)
  File "<string>", line 6, in sender
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 611, in sender
    sender._ewait(lambda: sender.linked)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 818, in _ewait
    result = self.session._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 585, in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 224, in _ewait
    self.check_error()
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 217, in check_error
    raise e
SchedulingError: Couldn't apply scheduled task babysit: connection aborted

# cat /etc/qpidd.conf 
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Configuration file for qpidd. Entries are of the form: 
#   name=value 
#
# (Note: no spaces on either side of '=').
# Run "qpidd --help" or see "man qpidd" for more details.

auth=no

-----
/etc/pulp/server.conf

[messaging]
url: tcp://localhost:5672
transport: qpid
auth_enabled: true
cacert: /etc/pki/qpid/ca/ca.crt
clientcert: /etc/pki/qpid/client/client.pem
topic_exchange: 'amq.topic'

# = Asynchronous Tasks =
#
# Controls Pulp's Celery settings.
#
# broker_url: A URL to a broker that Celery can use to queue tasks. For example,
# 	      qpid://<username>:<password>@<hostname>:<port>/<vhost>/
[tasks]
broker_url: qpid://guest@localhost/

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list