[Pulp-list] create remote to redhat on pulp 3

Fri Jun 28 19:44:48 UTC 2019

Please file an issue at https://pulp.plan.io/issues/new/ and we will try to
reproduce the issue.

On Thu, Jun 27, 2019, 5:50 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:

> Tried again for both rhel7 and rhel6 repo. The download_concurrency is
> always set to "1". We tried to sync rhel6 repo first and task was
> completed. Rerun sync on rhel7 repo failed again. Our network team didn't
> find any issues on the network and proxy.
>
> This is our remote setting:
> {
> "_created": "2019-06-21T16:10:05.493082Z",
> "_href":
> "/pulp/api/v3/remotes/rpm/rpm/b308a912-1dc9-46cb-b340-450a96e5ee93/",
> "_last_updated": "2019-06-21T16:10:05.493107Z",
> "_type": "rpm.rpm",
> "download_concurrency": 1,
> "name": "rhel-x86_64-server-7",
> "policy": "immediate",
> "proxy_url": "http://proxyhost.bloomberg.com:8000",
> "ssl_ca_certificate":
> "c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21",
> "ssl_client_certificate":
> "c5c5dfac3e16e93b394af137ebf8f0c5eaa37277699e72dc8042a53f6133b999",
> "ssl_client_key":
> "3f8bde4c2baf94196a9e40d0834f86a2890b3df08053eef9283b3c136aae487f",
> "ssl_validation": true,
> "url": "
> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/",
> "validate": true
> }
>
> Task status:
>
> Allow: GET, PATCH, DELETE, HEAD, OPTIONS
> Connection: close
> Content-Length: 3515
> Content-Type: application/json
> Date: Thu, 27 Jun 2019 21:32:27 GMT
> Server: gunicorn/19.9.0
> Vary: Accept, Cookie
> X-Frame-Options: SAMEORIGIN
>
> {
> "_created": "2019-06-27T18:15:28.484561Z",
> "_href": "/pulp/api/v3/tasks/72da60f0-1beb-4447-b8d2-718a40b5fccc/",
> "created_resources": [],
> "error": {
> "code": null,
> "description": "Timeout on reading data from socket",
> "traceback": " File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812,
> in perform_job\n rv = job.perform()\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
> perform\n self._result = self._execute()\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
> _execute\n return self.func(*self.args, **self.kwargs)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
> line 68, in synchronize\n dv.create()\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
> line 169, in create\n loop.run_until_complete(pipeline)\n File
> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
> run_until_complete\n return future.result()\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
> line 209, in create_pipeline\n await asyncio.gather(*futures)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
> line 43, in __call__\n await self.run()\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
> line 132, in run\n pb.done += task.result() # download_count\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
> line 155, in _handle_content_unit\n await
> asyncio.gather(*downloaders_for_content)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\",
> line 78, in download\n download_result = await
> downloader.run(extra_data=self.extra_data)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\",
> line 212, in run\n return await self._run(extra_data=extra_data)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line
> 131, in retry\n ret = await target(*args, **kwargs)\n File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
> line 184, in _run\n to_return = await self._handle_response(response)\n
> File
> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
> line 158, in _handle_response\n chunk = await
> response.content.read(1048576) # 1 megabyte\n File
> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\",
> line 369, in read\n await self._wait('read')\n File
> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\",
> line 297, in _wait\n await waiter\n"
> },
> "finished_at": "2019-06-27T21:19:18.675004Z",
> "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
> "non_fatal_errors": [],
> "parent": null,
> "progress_reports": [
> {
> "done": 20201,
> "message": "Parsed Packages",
> "state": "running",
> "suffix": null,
> "total": 24397
> },
> {
> "done": 11966,
> "message": "Downloading Artifacts",
> "state": "failed",
> "suffix": null,
> "total": 11966
> },
> {
> "done": 5,
> "message": "Downloading Metadata Files",
> "state": "canceled",
> "suffix": null,
> "total": 5
> },
> {
> "done": 23300,
> "message": "Associating Content",
> "state": "canceled",
> "suffix": null,
> "total": 23300
> },
> {
> "done": 3583,
> "message": "Parsed Erratum",
> "state": "running",
> "suffix": null,
> "total": 3583
> }
> ],
> "spawned_tasks": [],
> "started_at": "2019-06-27T18:15:28.618096Z",
> "state": "failed",
> "worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
> }
>
> From: bbouters at redhat.com At: 06/27/19 10:54:05
> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
> Cc: dkliban at redhat.com, pulp-list at redhat.com
> Subject: Re: [Pulp-list] create remote to redhat on pulp 3
>
> It says "Timeout on reading data from socket" so either this is a bug, or
> the remote server is not delivering you data over the socket connection,
> perhaps due to some rate limiting. The remote has a download_concurrency
> option [0] so perhaps reducing that will cause the server to give you all
> of the data. You can also open an issue with reproducer details and we can
> try to reproduce it.
>
> [0]:
> https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create
>
> On Thu, Jun 27, 2019 at 10:48 AM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>> I am now able to sync the redhat repo through proxy but the task failed
>> eventually on a timeout after sync most of contents. I tried sync both
>> rhel7 and rhel6 os repo and both gave the same error. Any idea what could
>> cause this?
>>
>>
>> # ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/
>> HTTP/1.1 200 OK
>> Allow: GET, PATCH, DELETE, HEAD, OPTIONS
>> Connection: close
>> Content-Length: 3512
>> Content-Type: application/json
>> Date: Thu, 27 Jun 2019 14:38:12 GMT
>> Server: gunicorn/19.9.0
>> Vary: Accept, Cookie
>> X-Frame-Options: SAMEORIGIN
>>
>> {
>> "_created": "2019-06-26T20:35:38.514762Z",
>> "_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/",
>> "created_resources": [],
>> "error": {
>> "code": null,
>> "description": "Timeout on reading data from socket",
>> "traceback": " File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812,
>> in perform_job\n rv = job.perform()\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
>> perform\n self._result = self._execute()\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
>> _execute\n return self.func(*self.args, **self.kwargs)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>> line 68, in synchronize\n dv.create()\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>> line 169, in create\n loop.run_until_complete(pipeline)\n File
>> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
>> run_until_complete\n return future.result()\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
>> line 209, in create_pipeline\n await asyncio.gather(*futures)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
>> line 43, in __call__\n await self.run()\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>> line 132, in run\n pb.done += task.result() # download_count\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>> line 155, in _handle_content_unit\n await
>> asyncio.gather(*downloaders_for_content)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\",
>> line 78, in download\n download_result = await
>> downloader.run(extra_data=self.extra_data)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\",
>> line 212, in run\n return await self._run(extra_data=extra_data)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line
>> 131, in retry\n ret = await target(*args, **kwargs)\n File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
>> line 184, in _run\n to_return = await self._handle_response(response)\n
>> File
>> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
>> line 158, in _handle_response\n chunk = await
>> response.content.read(1048576) # 1 megabyte\n File
>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\",
>> line 369, in read\n await self._wait('read')\n File
>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\",
>> line 297, in _wait\n await waiter\n"
>> },
>> "finished_at": "2019-06-26T21:57:33.499085Z",
>> "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
>> "non_fatal_errors": [],
>> "parent": null,
>> "progress_reports": [
>> {
>> "done": 3757,
>> "message": "Downloading Artifacts",
>> "state": "failed",
>> "suffix": null,
>> "total": 3757
>> },
>> {
>> "done": 5,
>> "message": "Downloading Metadata Files",
>> "state": "canceled",
>> "suffix": null,
>> "total": 5
>> },
>> {
>> "done": 11350,
>> "message": "Associating Content",
>> "state": "canceled",
>> "suffix": null,
>> "total": 11350
>> },
>> {
>> "done": 3582,
>> "message": "Parsed Erratum",
>> "state": "running",
>> "suffix": null,
>> "total": 3582
>> },
>> {
>> "done": 8201,
>> "message": "Parsed Packages",
>> "state": "running",
>> "suffix": null,
>> "total": 24392
>> }
>> ],
>> "spawned_tasks": [],
>> "started_at": "2019-06-26T20:35:38.650635Z",
>> "state": "failed",
>> "worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
>> }
>>
>>
>>
>> From: dkliban at redhat.com At: 06/23/19 07:14:50
>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>> Cc: pulp-list at redhat.com
>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3
>>
>> The problem is that the proxy setting is not being respected by Pulp. I
>> filed an issue about this[0] and also made a patch to fix it[1]. This patch
>> should be merged in the next 24 hours.
>>
>> If you cannot access cdn.redhat.com without a proxy, the error you got
>> makes sense. I was only able to reproduce the error message when I added an
>> incorrect entry for cdn.redhat.com in /etc/hosts.
>>
>>
>> [0] https://pulp.plan.io/issues/5011
>> [1] https://github.com/pulp/pulpcore-plugin/pull/107
>>
>> On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> Thanks. I am now getting a different error. Please advise
>>>
>>> "_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/",
>>> "created_resources": [],
>>> "error": {
>>> "code": null,
>>> "description": "Cannot connect to host cdn.redhat.com:443 ssl:None
>>> [Connect call failed ('23.65.16.251', 443)]",
>>> "traceback": " File
>>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812,
>>> in perform_job\n rv = job.perform()\n File
>>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
>>> perform\n self._result = self._execute()\n File
>>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
>>> _execute\n return self.func(*self.args, **self.kwargs)\n File
>>> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in
>>> synchronize\n dv.create()\n File
>>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\",
>>> line 169, in create\n loop.run_until_complete(pipeline)\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
>>> run_until_complete\n return future.result()\n File
>>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in
>>> create_pipeline\n await asyncio.gather(*futures)\n File
>>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in
>>> __call__\n await self.run()\n File
>>> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n
>>> result = await downloader.run()\n File
>>> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in
>>> run\n return await self._run(extra_data=extra_data)\n File
>>> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line
>>> 131, in retry\n ret = await target(*args, **kwargs)\n File
>>> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in
>>> _run\n async with self.session.get(self.url) as response:\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line
>>> 1005, in __aenter__\n self._resp = await self._coro\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line
>>> 476, in _request\n timeout=real_timeout\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
>>> line 522, in connect\n proto = await self._create_connection(req, traces,
>>> timeout)\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
>>> line 854, in _create_connection\n req, traces, timeout)\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
>>> line 992, in _create_direct_connection\n raise last_exc\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
>>> line 974, in _create_direct_connection\n req=req,
>>> client_error=client_error)\n File
>>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
>>> line 931, in _wrap_create_connection\n raise
>>> client_error(req.connection_key, exc) from exc\n"
>>> },
>>>
>>>
>>> I was able to connect using client cert through proxy
>>> # curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key
>>> /etc/pki/entitlement/8521692907269500331-key.pem --cert
>>> /etc/pki/entitlement/8521692907269500331.pem
>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
>>> * About to connect() to proxy ... port ... (#0)
>>> * Trying ...
>>> * Connected to ... () port ... (#0)
>>> * Establish HTTP proxy tunnel to cdn.redhat.com:443
>>> > CONNECT cdn.redhat.com:443 HTTP/1.1
>>> > Host: cdn.redhat.com:443
>>> > User-Agent: curl/7.29.0
>>> > Proxy-Connection: Keep-Alive
>>> >
>>> < HTTP/1.1 200 Connection established
>>> HTTP/1.1 200 Connection established
>>> <
>>>
>>> * Proxy replied OK to CONNECT request
>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb
>>>
>>>
>>> From: dkliban at redhat.com At: 06/20/19 16:34:40
>>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>>> Cc: pulp-list at redhat.com
>>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3
>>>
>>> You need to make sure that the following services are running:
>>> pulp-resource-manager, pulp-worker at 0, pulp-worker at 1 (at least 1 worker
>>> is needed).
>>>
>>> On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>> bli111 at bloomberg.net> wrote:
>>>
>>>> I installed the latest version of component. No more errors when try to
>>>> sync the repo but the task is in waiting state forever. Any idea what I
>>>> should check?
>>>>
>>>> "versions": [
>>>> {
>>>> "component": "pulpcore",
>>>> "version": "3.0.0rc3.dev0"
>>>> },
>>>> {
>>>> "component": "pulpcore-plugin",
>>>> "version": "0.1.0rc3.dev0"
>>>> },
>>>> {
>>>> "component": "pulp_rpm",
>>>> "version": "3.0.0b4.dev0"
>>>> }
>>>>
>>>>
>>>>
>>>> From: dkliban at redhat.com At: 06/18/19 13:42:01
>>>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>>>> Cc: pulp-list at redhat.com
>>>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3
>>>>
>>>> This issue has now been resolved and a fix is available on master
>>>> branch of pulpcore-plugin[0].
>>>>
>>>>
>>>> [0] https://github.com/pulp/pulpcore-plugin/tree/master/
>>>>
>>>> On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <dkliban at redhat.com>
>>>> wrote:
>>>>
>>>>> There is an open issue about this[0]. I recommend clicking the watch
>>>>> button so you get notifications about it. We should be fixing this soon.
>>>>>
>>>>> [0] https://pulp.plan.io/issues/4825
>>>>>
>>>>> On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>>>> bli111 at bloomberg.net> wrote:
>>>>>
>>>>>>
>>>>>> I was not able to sync a protected repo from redhat. The host running
>>>>>> pulp 3 subscribes to redhat directly.
>>>>>>
>>>>>> I first tried to use the following to create remote:
>>>>>> {
>>>>>> "name": "rhel-x86_64-server-7",
>>>>>> "url": "
>>>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
>>>>>> "ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem",
>>>>>> "ssl_client_certificate": "/etc/pki/consumer/cert.pem",
>>>>>> "ssl_client_key": "/etc/pki/consumer/key.pem",
>>>>>> "ssl_validation": true,
>>>>>> "proxy_url": "http://proxy:80",
>>>>>> "download_concurrency": 1,
>>>>>> "policy": "immediate"
>>>>>> }
>>>>>>
>>>>>> Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244
>>>>>> rq[19389]: pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no
>>>>>> start line (_ssl.c:3626)" when I tried to sync with http POST
>>>>>> :24817${REMOTE_HREF}sync/ repository=$REPO_HREF
>>>>>>
>>>>>> Then I tried to converted the pem file to a json format
>>>>>> {
>>>>>> "name": "rhel-x86_64-server-7",
>>>>>> "url": "
>>>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os",
>>>>>> "ssl_ca_certificate": "-----BEGIN
>>>>>> CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg
>>>>>> ...
>>>>>> "ssl_client_certificate": "-----BEGIN
>>>>>> CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ
>>>>>> ...
>>>>>> "ssl_client_key": "-----BEGIN RSA PRIVATE
>>>>>> KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee
>>>>>> ...
>>>>>> "ssl_validation": true,
>>>>>> "proxy_url": "http://proxy:80",
>>>>>> "download_concurrency": 1,
>>>>>> "policy": "immediate"
>>>>>> }
>>>>>>
>>>>>> I got this error when I tried to sync repo with the remote:
>>>>>> Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM
>>>>>> lib (_ssl.c:3503)
>>>>>>
>>>>>>
>>>>>> What is the correct json input for creating a remote to redhat? How
>>>>>> do I troubleshoot the sync issue? Thanks for your help.
>>>>>> _______________________________________________
>>>>>> Pulp-list mailing list
>>>>>> Pulp-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190628/01f95033/attachment.htm>