[Pulp-list] create remote to redhat on pulp 3
Bin Li (BLOOMBERG/ 120 PARK)
bli111 at bloomberg.net
Thu Jun 27 14:47:11 UTC 2019
I am now able to sync the redhat repo through proxy but the task failed eventually on a timeout after sync most of contents. I tried sync both rhel7 and rhel6 os repo and both gave the same error. Any idea what could cause this?
# ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 3512
Content-Type: application/json
Date: Thu, 27 Jun 2019 14:38:12 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_created": "2019-06-26T20:35:38.514762Z",
"_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/",
"created_resources": [],
"error": {
"code": null,
"description": "Timeout on reading data from socket",
"traceback": " File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n rv = job.perform()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in perform\n self._result = self._execute()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 68, in synchronize\n dv.create()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n return future.result()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n pb.done += task.result() # download_count\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 155, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 78, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", line 212, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131, in retry\n ret = await target(*args, **kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 184, in _run\n to_return = await self._handle_response(response)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 158, in _handle_response\n chunk = await response.content.read(1048576) # 1 megabyte\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 369, in read\n await self._wait('read')\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 297, in _wait\n await waiter\n"
},
"finished_at": "2019-06-26T21:57:33.499085Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 3757,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": 3757
},
{
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": 5
},
{
"done": 11350,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": 11350
},
{
"done": 3582,
"message": "Parsed Erratum",
"state": "running",
"suffix": null,
"total": 3582
},
{
"done": 8201,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"total": 24392
}
],
"spawned_tasks": [],
"started_at": "2019-06-26T20:35:38.650635Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
}
From: dkliban at redhat.com At: 06/23/19 07:14:50To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
The problem is that the proxy setting is not being respected by Pulp. I filed an issue about this[0] and also made a patch to fix it[1]. This patch should be merged in the next 24 hours.
If you cannot access cdn.redhat.com without a proxy, the error you got makes sense. I was only able to reproduce the error message when I added an incorrect entry for cdn.redhat.com in /etc/hosts.
[0] https://pulp.plan.io/issues/5011
[1] https://github.com/pulp/pulpcore-plugin/pull/107
On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Thanks. I am now getting a different error. Please advise
"_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/",
"created_resources": [],
"error": {
"code": null,
"description": "Cannot connect to host cdn.redhat.com:443 ssl:None [Connect call failed ('23.65.16.251', 443)]",
"traceback": " File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n rv = job.perform()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in perform\n self._result = self._execute()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in synchronize\n dv.create()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n return future.result()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n result = await downloader.run()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131, in retry\n ret = await target(*args, **kwargs)\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in _run\n async with self.session.get(self.url) as response:\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line 1005, in __aenter__\n self._resp = await self._coro\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line 476, in _request\n timeout=real_timeout\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 522, in connect\n proto = await self._create_connection(req, traces, timeout)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 854, in _create_connection\n req, traces, timeout)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 992, in _create_direct_connection\n raise last_exc\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 974, in _create_direct_connection\n req=req, client_error=client_error)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 931, in _wrap_create_connection\n raise client_error(req.connection_key, exc) from exc\n"
},
I was able to connect using client cert through proxy
# curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key /etc/pki/entitlement/8521692907269500331-key.pem --cert /etc/pki/entitlement/8521692907269500331.pem https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
* About to connect() to proxy ... port ... (#0)
* Trying ...
* Connected to ... () port ... (#0)
* Establish HTTP proxy tunnel to cdn.redhat.com:443
> CONNECT cdn.redhat.com:443 HTTP/1.1
> Host: cdn.redhat.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
From: dkliban at redhat.com At: 06/20/19 16:34:40To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
You need to make sure that the following services are running: pulp-resource-manager, pulp-worker at 0, pulp-worker at 1 (at least 1 worker is needed).
On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I installed the latest version of component. No more errors when try to sync the repo but the task is in waiting state forever. Any idea what I should check?
"versions": [
{
"component": "pulpcore",
"version": "3.0.0rc3.dev0"
},
{
"component": "pulpcore-plugin",
"version": "0.1.0rc3.dev0"
},
{
"component": "pulp_rpm",
"version": "3.0.0b4.dev0"
}
From: dkliban at redhat.com At: 06/18/19 13:42:01To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
This issue has now been resolved and a fix is available on master branch of pulpcore-plugin[0].
[0] https://github.com/pulp/pulpcore-plugin/tree/master/
On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <dkliban at redhat.com> wrote:
There is an open issue about this[0]. I recommend clicking the watch button so you get notifications about it. We should be fixing this soon.
[0] https://pulp.plan.io/issues/4825
On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I was not able to sync a protected repo from redhat. The host running pulp 3 subscribes to redhat directly.
I first tried to use the following to create remote:
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
"ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem",
"ssl_client_certificate": "/etc/pki/consumer/cert.pem",
"ssl_client_key": "/etc/pki/consumer/key.pem",
"ssl_validation": true,
"proxy_url": "http://proxy:80",
"download_concurrency": 1,
"policy": "immediate"
}
Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244 rq[19389]: pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no start line (_ssl.c:3626)" when I tried to sync with http POST :24817${REMOTE_HREF}sync/ repository=$REPO_HREF
Then I tried to converted the pem file to a json format
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
"ssl_ca_certificate": "-----BEGIN CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg
...
"ssl_client_certificate": "-----BEGIN CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ
...
"ssl_client_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee
...
"ssl_validation": true,
"proxy_url": "http://proxy:80",
"download_concurrency": 1,
"policy": "immediate"
}
I got this error when I tried to sync repo with the remote:
Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
What is the correct json input for creating a remote to redhat? How do I troubleshoot the sync issue? Thanks for your help._______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190627/8f9f7c01/attachment.htm>
More information about the Pulp-list
mailing list