[Pulp-list] pulp 3.7.3 aarch64 repo
Daniel Alley
dalley at redhat.com
Fri Jun 18 19:54:39 UTC 2021
Well.. the specific fix we need here *might* be backportable, although the
full patch is not. I can look into it next week, if the issues continue.
On Fri, Jun 18, 2021 at 3:46 PM Daniel Alley <dalley at redhat.com> wrote:
> Yup, that's the issue. I've seen all 3 of those errors before while I was
> testing my fix for this issue (retriable downloads). The cdn.redhat.com
> seems to be having a very bad day, today.
>
> It's a feature though, and not backportable, because the changes touch the
> database and several different areas of code. My only suggestions are to
> either reduce the download_concurrency further, use on_demand sync so that
> it doesn't immediately download packages at all, or wait (because the CDN
> issue is likely temporary). And upgrade once you get the chance (once
> Katello packages pulpcore 3.14).
>
> On Fri, Jun 18, 2021 at 3:03 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>> Still have errors on this repo
>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>
>>
>> Got 3 different errors for last 3 runs.
>> Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR:
>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>> completed
>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:17:22 pulpserver rq: dv.create()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:17:22 pulpserver rq: return future.result()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:17:22 pulpserver rq: await self.run()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:17:22 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:17:22 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:17:22 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 89, in _run
>> Jun 18 14:17:22 pulpserver rq: to_return = await
>> self._handle_response(response)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
>> line 180, in _handle_response
>> Jun 18 14:17:22 pulpserver rq: chunk = await
>> response.content.read(1048576) # 1 megabyte
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 368, in read
>> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 296, in _wait
>> Jun 18 14:17:22 pulpserver rq: await waiter
>> Jun 18 14:17:22 pulpserver rq:
>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>> completed
>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:17:22 pulpserver rq: dv.create()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:17:22 pulpserver rq: return future.result()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:17:22 pulpserver rq: await self.run()
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:17:22 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:17:22 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:17:22 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 89, in _run
>> Jun 18 14:17:22 pulpserver rq: to_return = await
>> self._handle_response(response)
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
>> line 180, in _handle_response
>> Jun 18 14:17:22 pulpserver rq: chunk = await
>> response.content.read(1048576) # 1 megabyte
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 368, in read
>> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
>> Jun 18 14:17:22 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 296, in _wait
>> Jun 18 14:17:22 pulpserver rq: await waiter
>> Jun 18 14:17:22 pulpserver rq:
>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>> completed
>> ==================================
>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 936, in _wrap_create_connection
>> Jun 18 14:34:09 pulpserver rq: return await
>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
>> create_connection
>> Jun 18 14:34:09 pulpserver rq:
>> ssl_handshake_timeout=ssl_handshake_timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
>> _create_connection_transport
>> Jun 18 14:34:09 pulpserver rq: await waiter
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
>> data_received
>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
>> self._sslpipe.feed_ssldata(data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
>> feed_ssldata
>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause
>> of the following exception:
>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:34:09 pulpserver rq: dv.create()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:34:09 pulpserver rq: return future.result()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:34:09 pulpserver rq: await self.run()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:34:09 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:34:09 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:34:09 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 87, in _run
>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
>> proxy=self.proxy, auth=self.auth) as response:
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 1012, in __aenter__
>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 483, in _request
>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 523, in connect
>> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req,
>> traces, timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 856, in _create_connection
>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 1097, in _create_proxy_connection
>> Jun 18 14:34:09 pulpserver rq: req=req)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 941, in _wrap_create_connection
>> Jun 18 14:34:09 pulpserver rq: raise
>> ClientConnectorSSLError(req.connection_key, exc) from exc
>> Jun 18 14:34:09 pulpserver rq:
>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
>> sslv3 alert unexpected message (_ssl.c:1056)]
>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 936, in _wrap_create_connection
>> Jun 18 14:34:09 pulpserver rq: return await
>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
>> create_connection
>> Jun 18 14:34:09 pulpserver rq:
>> ssl_handshake_timeout=ssl_handshake_timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
>> _create_connection_transport
>> Jun 18 14:34:09 pulpserver rq: await waiter
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
>> data_received
>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
>> self._sslpipe.feed_ssldata(data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
>> feed_ssldata
>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause
>> of the following exception:
>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:34:09 pulpserver rq: dv.create()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:34:09 pulpserver rq: return future.result()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:34:09 pulpserver rq: await self.run()
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:34:09 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:34:09 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:34:09 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 87, in _run
>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
>> proxy=self.proxy, auth=self.auth) as response:
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 1012, in __aenter__
>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 483, in _request
>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 523, in connect
>> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req,
>> traces, timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 856, in _create_connection
>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 1097, in _create_proxy_connection
>> Jun 18 14:34:09 pulpserver rq: req=req)
>> Jun 18 14:34:09 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>> line 941, in _wrap_create_connection
>> Jun 18 14:34:09 pulpserver rq: raise
>> ClientConnectorSSLError(req.connection_key, exc) from exc
>> Jun 18 14:34:09 pulpserver rq:
>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
>> sslv3 alert unexpected message (_ssl.c:1056)]
>>
>> =================================
>> Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR:
>> aiohttp.client_exceptions.ServerDisconnectedError
>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:47:28 pulpserver rq: dv.create()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:47:28 pulpserver rq: return future.result()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:47:28 pulpserver rq: await self.run()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:47:28 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:47:28 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:47:28 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 87, in _run
>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
>> proxy=self.proxy, auth=self.auth) as response:
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 1012, in __aenter__
>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 504, in _request
>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
>> line 847, in start
>> Jun 18 14:47:28 pulpserver rq: message, payload = await
>> self._protocol.read() # type: ignore # noqa
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 591, in read
>> Jun 18 14:47:28 pulpserver rq: await self._waiter
>> Jun 18 14:47:28 pulpserver rq:
>> aiohttp.client_exceptions.ServerDisconnectedError
>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>> line 886, in perform_job
>> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 664, in perform
>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>> 670, in _execute
>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>> line 266, in synchronize
>> Jun 18 14:47:28 pulpserver rq: dv.create()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>> line 148, in create
>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>> run_until_complete
>> Jun 18 14:47:28 pulpserver rq: return future.result()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 225, in create_pipeline
>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>> line 43, in __call__
>> Jun 18 14:47:28 pulpserver rq: await self.run()
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 152, in run
>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>> line 178, in _handle_content_unit
>> Jun 18 14:47:28 pulpserver rq: await
>> asyncio.gather(*downloaders_for_content)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>> line 88, in download
>> Jun 18 14:47:28 pulpserver rq: download_result = await
>> downloader.run(extra_data=self.extra_data)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>> line 227, in run
>> Jun 18 14:47:28 pulpserver rq: return await
>> self._run(extra_data=extra_data)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>> line 87, in _run
>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
>> proxy=self.proxy, auth=self.auth) as response:
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 1012, in __aenter__
>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>> line 504, in _request
>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
>> line 847, in start
>> Jun 18 14:47:28 pulpserver rq: message, payload = await
>> self._protocol.read() # type: ignore # noqa
>> Jun 18 14:47:28 pulpserver rq: File
>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>> line 591, in read
>> Jun 18 14:47:28 pulpserver rq: await self._waiter
>> Jun 18 14:47:28 pulpserver rq:
>> aiohttp.client_exceptions.ServerDisconnectedError
>>
>>
>> The aarch64 baseos repo completed successfully.
>>
>>
>>
>>
>> From: dalley at redhat.com At: 06/18/21 10:55:56 UTC-4:00
>> To: dkliban at redhat.com
>> Cc: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>,
>> pulp-list at redhat.com
>> Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo
>>
>> replace "but only with the errors" with "but only with the normal types
>> of errors"
>>
>> On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dalley at redhat.com> wrote:
>>
>>> @Bin Li <bli111 at bloomberg.net>
>>> Could you post the error that was logged? I think this traceback is
>>> missing some of the relevant details.
>>>
>>> I can tell you what I expect that problem *might* be, because we've been
>>> hearing about it a lot recently. Again, this is just a guess, without more
>>> information.
>>>
>>> We have found that some content delivery networks (including the Red Hat
>>> one) start returning seemingly irrelevant error messages when they get
>>> 'angry' about too much traffic coming from one source. Rather than
>>> returning HTTP 429 or some other standard response, sometimes it will just
>>> terminate the connection, or give you SSL errors, or return an incomplete
>>> payload. This might be some type of DDoS prevention / flood control
>>> mechanism, I'm not entirely sure. The mitigation is to either reduce the
>>> "download_concurrency" on your remote from 10 to 5, or using "on_demand"
>>> syncs rather than "immediate". Both of those are just to lessen the load on
>>> the CDN and make it less likely to start dropping connections.
>>>
>>> But the "proper" fix is being worked on for pulpcore 3.14. We want to
>>> automatically retry downloads that fail using a backoff algorithm to
>>> prevent the entire sync from failing. We were already doing this, but only
>>> with the errors, rather than the weird ones that seem to come up in
>>> practice. And we're making the number of retry attempts configurable.
>>>
>>> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com>
>>> wrote:
>>>
>>>> I don't see anything special about these repos. Are you still
>>>> experiencing this problem?
>>>>
>>>>
>>>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>>> bli111 at bloomberg.net> wrote:
>>>>
>>>>> Tried to download aarch64 repo and got the errors below:
>>>>>
>>>>> "description": "",
>>>>> "traceback": " File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
>>>>> line 886, in perform_job\n rv = job.perform()\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>>> line 664, in perform\n self._result = self._execute()\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>>>>> line 266, in synchronize\n dv.create()\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>>>>> line 148, in create\n loop.run_until_complete(pipeline)\n File
>>>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
>>>>> run_until_complete\n return future.result()\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>>> line 43, in __call__\n await self.run()\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>>> line 152, in run\n pb.done += task.result() # download_count\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>>> line 178, in _handle_content_unit\n await
>>>>> asyncio.gather(*downloaders_for_content)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
>>>>> line 88, in download\n download_result = await
>>>>> downloader.run(extra_data=self.extra_data)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
>>>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
>>>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
>>>>> auth=self.auth) as response:\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>>> line 1012, in __aenter__\n self._resp = await self._coro\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>>> line 504, in _request\n await resp.start(conn)\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
>>>>> line 847, in start\n message, payload = await self._protocol.read() # type:
>>>>> ignore # noqa\n File
>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
>>>>> line 591, in read\n await self._waiter\n
>>>>>
>>>>> The same error happened on both remote
>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>>>>
>>>>> Please advise.
>>>>>
>>>>> Thanks
>>>>> _______________________________________________
>>>>> Pulp-list mailing list
>>>>> Pulp-list at redhat.com
>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210618/4f4e11d5/attachment.htm>
More information about the Pulp-list
mailing list