[Pulp-list] create remote to redhat on pulp 3
Bin Li (BLOOMBERG/ 120 PARK)
bli111 at bloomberg.net
Thu Jun 27 21:49:52 UTC 2019
Tried again for both rhel7 and rhel6 repo. The download_concurrency is always set to "1". We tried to sync rhel6 repo first and task was completed. Rerun sync on rhel7 repo failed again. Our network team didn't find any issues on the network and proxy.
This is our remote setting:
{
"_created": "2019-06-21T16:10:05.493082Z",
"_href": "/pulp/api/v3/remotes/rpm/rpm/b308a912-1dc9-46cb-b340-450a96e5ee93/",
"_last_updated": "2019-06-21T16:10:05.493107Z",
"_type": "rpm.rpm",
"download_concurrency": 1,
"name": "rhel-x86_64-server-7",
"policy": "immediate",
"proxy_url": "http://proxyhost.bloomberg.com:8000",
"ssl_ca_certificate": "c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21",
"ssl_client_certificate": "c5c5dfac3e16e93b394af137ebf8f0c5eaa37277699e72dc8042a53f6133b999",
"ssl_client_key": "3f8bde4c2baf94196a9e40d0834f86a2890b3df08053eef9283b3c136aae487f",
"ssl_validation": true,
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/",
"validate": true
}
Task status:
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 3515
Content-Type: application/json
Date: Thu, 27 Jun 2019 21:32:27 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_created": "2019-06-27T18:15:28.484561Z",
"_href": "/pulp/api/v3/tasks/72da60f0-1beb-4447-b8d2-718a40b5fccc/",
"created_resources": [],
"error": {
"code": null,
"description": "Timeout on reading data from socket",
"traceback": " File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n rv = job.perform()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in perform\n self._result = self._execute()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 68, in synchronize\n dv.create()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n return future.result()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n pb.done += task.result() # download_count\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 155, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 78, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", line 212, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131, in retry\n ret = await target(*args, **kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 184, in _run\n to_return = await self._handle_response(response)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 158, in _handle_response\n chunk = await response.content.read(1048576) # 1 megabyte\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 369, in read\n await self._wait('read')\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 297, in _wait\n await waiter\n"
},
"finished_at": "2019-06-27T21:19:18.675004Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 20201,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"total": 24397
},
{
"done": 11966,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": 11966
},
{
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": 5
},
{
"done": 23300,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": 23300
},
{
"done": 3583,
"message": "Parsed Erratum",
"state": "running",
"suffix": null,
"total": 3583
}
],
"spawned_tasks": [],
"started_at": "2019-06-27T18:15:28.618096Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
}
From: bbouters at redhat.com At: 06/27/19 10:54:05To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: dkliban at redhat.com, pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
It says "Timeout on reading data from socket" so either this is a bug, or the remote server is not delivering you data over the socket connection, perhaps due to some rate limiting. The remote has a download_concurrency option [0] so perhaps reducing that will cause the server to give you all of the data. You can also open an issue with reproducer details and we can try to reproduce it.
[0]: https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create
On Thu, Jun 27, 2019 at 10:48 AM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I am now able to sync the redhat repo through proxy but the task failed eventually on a timeout after sync most of contents. I tried sync both rhel7 and rhel6 os repo and both gave the same error. Any idea what could cause this?
# ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 3512
Content-Type: application/json
Date: Thu, 27 Jun 2019 14:38:12 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_created": "2019-06-26T20:35:38.514762Z",
"_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/",
"created_resources": [],
"error": {
"code": null,
"description": "Timeout on reading data from socket",
"traceback": " File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n rv = job.perform()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in perform\n self._result = self._execute()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 68, in synchronize\n dv.create()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n return future.result()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 132, in run\n pb.done += task.result() # download_count\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 155, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 78, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", line 212, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131, in retry\n ret = await target(*args, **kwargs)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 184, in _run\n to_return = await self._handle_response(response)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", line 158, in _handle_response\n chunk = await response.content.read(1048576) # 1 megabyte\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 369, in read\n await self._wait('read')\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line 297, in _wait\n await waiter\n"
},
"finished_at": "2019-06-26T21:57:33.499085Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 3757,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": 3757
},
{
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": 5
},
{
"done": 11350,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": 11350
},
{
"done": 3582,
"message": "Parsed Erratum",
"state": "running",
"suffix": null,
"total": 3582
},
{
"done": 8201,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"total": 24392
}
],
"spawned_tasks": [],
"started_at": "2019-06-26T20:35:38.650635Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
}
From: dkliban at redhat.com At: 06/23/19 07:14:50To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
The problem is that the proxy setting is not being respected by Pulp. I filed an issue about this[0] and also made a patch to fix it[1]. This patch should be merged in the next 24 hours.
If you cannot access cdn.redhat.com without a proxy, the error you got makes sense. I was only able to reproduce the error message when I added an incorrect entry for cdn.redhat.com in /etc/hosts.
[0] https://pulp.plan.io/issues/5011
[1] https://github.com/pulp/pulpcore-plugin/pull/107
On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Thanks. I am now getting a different error. Please advise
"_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/",
"created_resources": [],
"error": {
"code": null,
"description": "Cannot connect to host cdn.redhat.com:443 ssl:None [Connect call failed ('23.65.16.251', 443)]",
"traceback": " File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in perform_job\n rv = job.perform()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in perform\n self._result = self._execute()\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in synchronize\n dv.create()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\n return future.result()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n result = await downloader.run()\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131, in retry\n ret = await target(*args, **kwargs)\n File \"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in _run\n async with self.session.get(self.url) as response:\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line 1005, in __aenter__\n self._resp = await self._coro\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line 476, in _request\n timeout=real_timeout\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 522, in connect\n proto = await self._create_connection(req, traces, timeout)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 854, in _create_connection\n req, traces, timeout)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 992, in _create_direct_connection\n raise last_exc\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 974, in _create_direct_connection\n req=req, client_error=client_error)\n File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line 931, in _wrap_create_connection\n raise client_error(req.connection_key, exc) from exc\n"
},
I was able to connect using client cert through proxy
# curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key /etc/pki/entitlement/8521692907269500331-key.pem --cert /etc/pki/entitlement/8521692907269500331.pem https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
* About to connect() to proxy ... port ... (#0)
* Trying ...
* Connected to ... () port ... (#0)
* Establish HTTP proxy tunnel to cdn.redhat.com:443
> CONNECT cdn.redhat.com:443 HTTP/1.1
> Host: cdn.redhat.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
From: dkliban at redhat.com At: 06/20/19 16:34:40To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
You need to make sure that the following services are running: pulp-resource-manager, pulp-worker at 0, pulp-worker at 1 (at least 1 worker is needed).
On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I installed the latest version of component. No more errors when try to sync the repo but the task is in waiting state forever. Any idea what I should check?
"versions": [
{
"component": "pulpcore",
"version": "3.0.0rc3.dev0"
},
{
"component": "pulpcore-plugin",
"version": "0.1.0rc3.dev0"
},
{
"component": "pulp_rpm",
"version": "3.0.0b4.dev0"
}
From: dkliban at redhat.com At: 06/18/19 13:42:01To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
This issue has now been resolved and a fix is available on master branch of pulpcore-plugin[0].
[0] https://github.com/pulp/pulpcore-plugin/tree/master/
On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <dkliban at redhat.com> wrote:
There is an open issue about this[0]. I recommend clicking the watch button so you get notifications about it. We should be fixing this soon.
[0] https://pulp.plan.io/issues/4825
On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
I was not able to sync a protected repo from redhat. The host running pulp 3 subscribes to redhat directly.
I first tried to use the following to create remote:
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
"ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem",
"ssl_client_certificate": "/etc/pki/consumer/cert.pem",
"ssl_client_key": "/etc/pki/consumer/key.pem",
"ssl_validation": true,
"proxy_url": "http://proxy:80",
"download_concurrency": 1,
"policy": "immediate"
}
Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244 rq[19389]: pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no start line (_ssl.c:3626)" when I tried to sync with http POST :24817${REMOTE_HREF}sync/ repository=$REPO_HREF
Then I tried to converted the pem file to a json format
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
"ssl_ca_certificate": "-----BEGIN CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg
...
"ssl_client_certificate": "-----BEGIN CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ
...
"ssl_client_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee
...
"ssl_validation": true,
"proxy_url": "http://proxy:80",
"download_concurrency": 1,
"policy": "immediate"
}
I got this error when I tried to sync repo with the remote:
Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
What is the correct json input for creating a remote to redhat? How do I troubleshoot the sync issue? Thanks for your help._______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190627/9ec60bea/attachment.htm>
More information about the Pulp-list
mailing list