<div dir="ltr">This problem might be related to this bug: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1129584">https://bugzilla.redhat.com/show_bug.cgi?id=1129584</a></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 15, 2014 at 11:25 AM, Cristian Falcas <span dir="ltr"><<a href="mailto:cristi.falcas@gmail.com" target="_blank">cristi.falcas@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>We get the same error when trying to sync or delete a puppet local repo (also el7):<br><br>2014-09-14 16:40:27,016 - INFO - DELETE request to /pulp/api/v2/repositories/puppet-builds/ with parameters None<br>2014-09-14 16:40:27,017 - INFO - Response status : 500<br><br>2014-09-14 16:40:27,017 - INFO - Response body :<br> {<br> "exception": [<br> "AuthenticationFailure: connection-forced: Authentication failed(320)\n"<br> ],<br> "traceback": [<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py\", line 44, in __call__\n return self.app(environ, start_response)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py\", line 42, in __call__\n return self.app(environ, start_response)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 279, in wsgi\n result = self.handle_with_processors()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 29, in _handle_with_processors\n return process(self.processors)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 566, in processor\n return handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 581, in processor\n result = handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 28, in process\n return self.handle()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 230, in handle\n return self._delegate(fn, self.fvars, args)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 422, in _delegate\n return f()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 430, in <lambda>\n f = lambda: self._delegate_sub_application(pat, what)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 455, in _delegate_sub_application\n return app.handle_with_processors()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 29, in _handle_with_processors\n return process(self.processors)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 566, in processor\n return handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 581, in processor\n result = handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 28, in process\n return self.handle()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 230, in handle\n return self._delegate(fn, self.fvars, args)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 420, in _delegate\n return handle_class(cls)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 396, in handle_class\n return tocall(*args)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py\", line 227, in _auth_decorator\n value = method(self, *args, **kwargs)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/repositories.py\", line 257, in DELETE\n [repo_id], tags=task_tags)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 234, in apply_async_with_reservation\n queue = _reserve_resource.apply_async((resource_id,), queue=RESOURCE_MANAGER_QUEUE).get()\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 555, in apply_async\n **dict(self._get_exec_options(), **options)\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 351, in send_task\n reply_to=reply_to or self.oid, **options\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 305, in publish_task\n **kwargs\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 168, in publish\n routing_key, mandatory, immediate, exchange, declare)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 440, in _ensured\n return fun(*args, **kwargs)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 173, in _publish\n channel = self.channel\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 190, in _get_channel\n channel = self._channel = channel()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py\", line 422, in __call__\n value = self.__value__ = self.__contract__()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 205, in <lambda>\n channel = ChannelPromise(lambda: connection.default_channel)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 773, in default_channel\n self.connection\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 758, in connection\n self._connection = self._establish_connection()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 713, in _establish_connection\n conn = self.transport.establish_connection()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1673, in establish_connection\n conn = self.Connection(**opts)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1365, in __init__\n self._qpid_conn = establish(**self.connection_options)\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 68, in establish\n conn.open(timeout=timeout)\n",<br> " File \"<string>\", line 6, in open\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 268, in open\n self.attach(timeout=timeout)\n",<br> " File \"<string>\", line 6, in attach\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 286, in attach\n if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 219, in _ewait\n self.check_error()\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 212, in check_error\n raise e\n"<br> ],<br> "_href": "/pulp/api/v2/repositories/puppet-builds/",<br> "error_message": "Unhandled Exception",<br> "http_request_method": "DELETE",<br> "http_status": 500<br>}<br><br><br><br></div>and from sync:<br><br>2014-09-14 16:32:00,422 - INFO - POST request to /pulp/api/v2/tasks/search/ with parameters {"criteria": {"filters": {"state": {"$nin": ["finished", "error", "canceled", "skipped"]}, "tags": {"$all": ["pulp:repository:testes", "pulp:action:sync"]}}}}<br>2014-09-14 16:32:00,422 - INFO - Response status : 200<br><br>2014-09-14 16:32:00,422 - INFO - Response body :<br> []<br><br>2014-09-14 16:32:00,601 - INFO - POST request to /pulp/api/v2/repositories/testes/actions/sync/ with parameters {"override_config": null}<br>2014-09-14 16:32:00,601 - INFO - Response status : 500<br><br>2014-09-14 16:32:00,602 - INFO - Response body :<br> {<br> "exception": [<br> "AuthenticationFailure: connection-forced: Authentication failed(320)\n"<br> ],<br> "traceback": [<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py\", line 44, in __call__\n return self.app(environ, start_response)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py\", line 42, in __call__\n return self.app(environ, start_response)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 279, in wsgi\n result = self.handle_with_processors()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 29, in _handle_with_processors\n return process(self.processors)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 566, in processor\n return handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 581, in processor\n result = handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 28, in process\n return self.handle()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 230, in handle\n return self._delegate(fn, self.fvars, args)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 422, in _delegate\n return f()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 430, in <lambda>\n f = lambda: self._delegate_sub_application(pat, what)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 455, in _delegate_sub_application\n return app.handle_with_processors()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 29, in _handle_with_processors\n return process(self.processors)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 566, in processor\n return handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in process\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 581, in processor\n result = handler()\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 26, in <lambda>\n return p(lambda: process(processors))\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py\", line 28, in process\n return self.handle()\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 230, in handle\n return self._delegate(fn, self.fvars, args)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 420, in _delegate\n return handle_class(cls)\n",<br> " File \"/usr/lib/python2.7/site-packages/web/application.py\", line 396, in handle_class\n return tocall(*args)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py\", line 227, in _auth_decorator\n value = method(self, *args, **kwargs)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/repositories.py\", line 710, in POST\n repo_id, [repo_id, overrides], {}, tags=task_tags)\n",<br> " File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 234, in apply_async_with_reservation\n queue = _reserve_resource.apply_async((resource_id,), queue=RESOURCE_MANAGER_QUEUE).get()\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 555, in apply_async\n **dict(self._get_exec_options(), **options)\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 351, in send_task\n reply_to=reply_to or self.oid, **options\n",<br> " File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 305, in publish_task\n **kwargs\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 168, in publish\n routing_key, mandatory, immediate, exchange, declare)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 440, in _ensured\n return fun(*args, **kwargs)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 173, in _publish\n channel = self.channel\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 190, in _get_channel\n channel = self._channel = channel()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py\", line 422, in __call__\n value = self.__value__ = self.__contract__()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 205, in <lambda>\n channel = ChannelPromise(lambda: connection.default_channel)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 773, in default_channel\n self.connection\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 758, in connection\n self._connection = self._establish_connection()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/connection.py\", line 713, in _establish_connection\n conn = self.transport.establish_connection()\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1673, in establish_connection\n conn = self.Connection(**opts)\n",<br> " File \"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py\", line 1365, in __init__\n self._qpid_conn = establish(**self.connection_options)\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 68, in establish\n conn.open(timeout=timeout)\n",<br> " File \"<string>\", line 6, in open\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 268, in open\n self.attach(timeout=timeout)\n",<br> " File \"<string>\", line 6, in attach\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 286, in attach\n if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 219, in _ewait\n self.check_error()\n",<br> " File \"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py\", line 212, in check_error\n raise e\n"<br> ],<br> "_href": "/pulp/api/v2/repositories/testes/actions/sync/",<br> "error_message": "Unhandled Exception",<br> "http_request_method": "POST",<br> "http_status": 500<br>}<br><br><br><div><div><br><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Sep 15, 2014 at 12:11 PM, johan petersson <span dir="ltr"><<a href="mailto:j.petersson@gmail.com" target="_blank">j.petersson@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">Hi,</span><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">I can't seem to get sync to a child node to work.</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">It is a new fresh installation - no upgrade.</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">I followed all the instructions for Pulp Server and Consumer installation and that part works.</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">However when using the instructions here for the node configuration: <a href="http://pulp-user-guide.readthedocs.org/en/latest/nodes.html" target="_blank">http://pulp-user-guide.readthedocs.org/en/latest/nodes.html</a><a href="http://pulp-user-guide.readthedocs.org/en/latest/nodes.html" style="font-size:10pt" target="_blank">http://pulp-user-guide.readthedocs.org/en/latest/nodes.html</a></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">Node synchronization fails with the following error message: -01: An unexpected error occurred. repo_id=None</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">I have set oauth in both server.conf and nodes.conf correctly, using default certs everywhere and have set verify_ssl: False.</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">I have even tried new installation on new Vm's but get the same result.</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px">Here is some info from the log:</div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><span style="font-size:10pt">Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: </span><span style="font-size:10pt">File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 192, in _auth_decorator</span></div><div style="color:rgb(0,0,0);font-family:Tahoma;font-size:13px"><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: userid, is_consumer = authenticate_user()</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 111, in oauth_authentication</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT)</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 88, in __init__ self.importers = self._importers() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET(path) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None</p><p style="margin-top:0px;margin-bottom:0px">Sep 15 10:42:05 repo goferd: [INFO][worker-0] gofer.agent.rmi:149 - sn=7b3315f2-ec18-4ced-be8b-2d714ae0b73b processed in: 114 (ms)</p><p style="margin-top:0px;margin-bottom:0px"><br></p><p style="margin-top:0px;margin-bottom:0px">It seems clear that it is a certificate error but i use verify_ssl: False </p><p style="margin-top:0px;margin-bottom:0px"><br></p><p style="margin-top:0px;margin-bottom:0px">I have also tried: pulp-consumer update --keys and restarted httpd and goferd with same result.</p><p style="margin-top:0px;margin-bottom:0px"><br></p><p style="margin-top:0px;margin-bottom:0px">I am now out of ideas, anyone here have any idea how to fix this problem?</p><p style="margin-top:0px;margin-bottom:0px"><br></p><p style="margin-top:0px;margin-bottom:0px">Thank you for any help,</p><p style="margin-top:0px;margin-bottom:0px"><br></p><p style="margin-top:0px;margin-bottom:0px">Johan</p></div></div> <br></div></div>_______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a><br></blockquote></div><br></div> </blockquote></div><br></div>