[Pulp-list] pulp 3.7.3 aarch64 repo
Daniel Alley
dalley at redhat.com
Fri Jun 18 19:46:04 UTC 2021
Yup, that's the issue. I've seen all 3 of those errors before while I was
testing my fix for this issue (retriable downloads). The cdn.redhat.com
seems to be having a very bad day, today.
It's a feature though, and not backportable, because the changes touch the
database and several different areas of code. My only suggestions are to
either reduce the download_concurrency further, use on_demand sync so that
it doesn't immediately download packages at all, or wait (because the CDN
issue is likely temporary). And upgrade once you get the chance (once
Katello packages pulpcore 3.14).
On Fri, Jun 18, 2021 at 3:03 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:
> Still have errors on this repo
> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>
>
> Got 3 different errors for last 3 runs.
> Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR:
> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
> completed
> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:17:22 pulpserver rq: dv.create()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:17:22 pulpserver rq: return future.result()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:17:22 pulpserver rq: await self.run()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:17:22 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:17:22 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:17:22 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 89, in _run
> Jun 18 14:17:22 pulpserver rq: to_return = await
> self._handle_response(response)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
> line 180, in _handle_response
> Jun 18 14:17:22 pulpserver rq: chunk = await
> response.content.read(1048576) # 1 megabyte
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 368, in read
> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 296, in _wait
> Jun 18 14:17:22 pulpserver rq: await waiter
> Jun 18 14:17:22 pulpserver rq:
> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
> completed
> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:17:22 pulpserver rq: dv.create()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:17:22 pulpserver rq: return future.result()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:17:22 pulpserver rq: await self.run()
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:17:22 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:17:22 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:17:22 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 89, in _run
> Jun 18 14:17:22 pulpserver rq: to_return = await
> self._handle_response(response)
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
> line 180, in _handle_response
> Jun 18 14:17:22 pulpserver rq: chunk = await
> response.content.read(1048576) # 1 megabyte
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 368, in read
> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
> Jun 18 14:17:22 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 296, in _wait
> Jun 18 14:17:22 pulpserver rq: await waiter
> Jun 18 14:17:22 pulpserver rq:
> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
> completed
> ==================================
> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 936, in _wrap_create_connection
> Jun 18 14:34:09 pulpserver rq: return await
> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
> create_connection
> Jun 18 14:34:09 pulpserver rq: ssl_handshake_timeout=ssl_handshake_timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
> _create_connection_transport
> Jun 18 14:34:09 pulpserver rq: await waiter
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
> data_received
> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
> self._sslpipe.feed_ssldata(data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
> feed_ssldata
> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause of
> the following exception:
> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:34:09 pulpserver rq: dv.create()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:34:09 pulpserver rq: return future.result()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:34:09 pulpserver rq: await self.run()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:34:09 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:34:09 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:34:09 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 87, in _run
> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
> proxy=self.proxy, auth=self.auth) as response:
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 1012, in __aenter__
> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 483, in _request
> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 523, in connect
> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req,
> traces, timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 856, in _create_connection
> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 1097, in _create_proxy_connection
> Jun 18 14:34:09 pulpserver rq: req=req)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 941, in _wrap_create_connection
> Jun 18 14:34:09 pulpserver rq: raise
> ClientConnectorSSLError(req.connection_key, exc) from exc
> Jun 18 14:34:09 pulpserver rq:
> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
> sslv3 alert unexpected message (_ssl.c:1056)]
> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 936, in _wrap_create_connection
> Jun 18 14:34:09 pulpserver rq: return await
> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
> create_connection
> Jun 18 14:34:09 pulpserver rq: ssl_handshake_timeout=ssl_handshake_timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
> _create_connection_transport
> Jun 18 14:34:09 pulpserver rq: await waiter
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
> data_received
> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
> self._sslpipe.feed_ssldata(data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
> feed_ssldata
> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause of
> the following exception:
> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:34:09 pulpserver rq: dv.create()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:34:09 pulpserver rq: return future.result()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:34:09 pulpserver rq: await self.run()
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:34:09 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:34:09 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:34:09 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 87, in _run
> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
> proxy=self.proxy, auth=self.auth) as response:
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 1012, in __aenter__
> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 483, in _request
> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 523, in connect
> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req,
> traces, timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 856, in _create_connection
> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 1097, in _create_proxy_connection
> Jun 18 14:34:09 pulpserver rq: req=req)
> Jun 18 14:34:09 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
> line 941, in _wrap_create_connection
> Jun 18 14:34:09 pulpserver rq: raise
> ClientConnectorSSLError(req.connection_key, exc) from exc
> Jun 18 14:34:09 pulpserver rq:
> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
> sslv3 alert unexpected message (_ssl.c:1056)]
>
> =================================
> Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR:
> aiohttp.client_exceptions.ServerDisconnectedError
> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:47:28 pulpserver rq: dv.create()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:47:28 pulpserver rq: return future.result()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:47:28 pulpserver rq: await self.run()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:47:28 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:47:28 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:47:28 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 87, in _run
> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
> proxy=self.proxy, auth=self.auth) as response:
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 1012, in __aenter__
> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 504, in _request
> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
> line 847, in start
> Jun 18 14:47:28 pulpserver rq: message, payload = await
> self._protocol.read() # type: ignore # noqa
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 591, in read
> Jun 18 14:47:28 pulpserver rq: await self._waiter
> Jun 18 14:47:28 pulpserver rq:
> aiohttp.client_exceptions.ServerDisconnectedError
> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
> line 886, in perform_job
> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 664, in perform
> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
> 670, in _execute
> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
> line 266, in synchronize
> Jun 18 14:47:28 pulpserver rq: dv.create()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
> line 148, in create
> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
> run_until_complete
> Jun 18 14:47:28 pulpserver rq: return future.result()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 225, in create_pipeline
> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
> line 43, in __call__
> Jun 18 14:47:28 pulpserver rq: await self.run()
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 152, in run
> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
> line 178, in _handle_content_unit
> Jun 18 14:47:28 pulpserver rq: await
> asyncio.gather(*downloaders_for_content)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
> line 88, in download
> Jun 18 14:47:28 pulpserver rq: download_result = await
> downloader.run(extra_data=self.extra_data)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
> line 227, in run
> Jun 18 14:47:28 pulpserver rq: return await
> self._run(extra_data=extra_data)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
> line 87, in _run
> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
> proxy=self.proxy, auth=self.auth) as response:
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 1012, in __aenter__
> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
> line 504, in _request
> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
> line 847, in start
> Jun 18 14:47:28 pulpserver rq: message, payload = await
> self._protocol.read() # type: ignore # noqa
> Jun 18 14:47:28 pulpserver rq: File
> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
> line 591, in read
> Jun 18 14:47:28 pulpserver rq: await self._waiter
> Jun 18 14:47:28 pulpserver rq:
> aiohttp.client_exceptions.ServerDisconnectedError
>
>
> The aarch64 baseos repo completed successfully.
>
>
>
>
> From: dalley at redhat.com At: 06/18/21 10:55:56 UTC-4:00
> To: dkliban at redhat.com
> Cc: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>,
> pulp-list at redhat.com
> Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo
>
> replace "but only with the errors" with "but only with the normal types of
> errors"
>
> On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dalley at redhat.com> wrote:
>
>> @Bin Li <bli111 at bloomberg.net>
>> Could you post the error that was logged? I think this traceback is
>> missing some of the relevant details.
>>
>> I can tell you what I expect that problem *might* be, because we've been
>> hearing about it a lot recently. Again, this is just a guess, without more
>> information.
>>
>> We have found that some content delivery networks (including the Red Hat
>> one) start returning seemingly irrelevant error messages when they get
>> 'angry' about too much traffic coming from one source. Rather than
>> returning HTTP 429 or some other standard response, sometimes it will just
>> terminate the connection, or give you SSL errors, or return an incomplete
>> payload. This might be some type of DDoS prevention / flood control
>> mechanism, I'm not entirely sure. The mitigation is to either reduce the
>> "download_concurrency" on your remote from 10 to 5, or using "on_demand"
>> syncs rather than "immediate". Both of those are just to lessen the load on
>> the CDN and make it less likely to start dropping connections.
>>
>> But the "proper" fix is being worked on for pulpcore 3.14. We want to
>> automatically retry downloads that fail using a backoff algorithm to
>> prevent the entire sync from failing. We were already doing this, but only
>> with the errors, rather than the weird ones that seem to come up in
>> practice. And we're making the number of retry attempts configurable.
>>
>> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com> wrote:
>>
>>> I don't see anything special about these repos. Are you still
>>> experiencing this problem?
>>>
>>>
>>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>> bli111 at bloomberg.net> wrote:
>>>
>>>> Tried to download aarch64 repo and got the errors below:
>>>>
>>>> "description": "",
>>>> "traceback": " File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
>>>> line 886, in perform_job\n rv = job.perform()\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>> line 664, in perform\n self._result = self._execute()\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>>>> line 266, in synchronize\n dv.create()\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>>>> line 148, in create\n loop.run_until_complete(pipeline)\n File
>>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
>>>> run_until_complete\n return future.result()\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>> line 43, in __call__\n await self.run()\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>> line 152, in run\n pb.done += task.result() # download_count\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>> line 178, in _handle_content_unit\n await
>>>> asyncio.gather(*downloaders_for_content)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
>>>> line 88, in download\n download_result = await
>>>> downloader.run(extra_data=self.extra_data)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
>>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
>>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
>>>> auth=self.auth) as response:\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>> line 1012, in __aenter__\n self._resp = await self._coro\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>> line 504, in _request\n await resp.start(conn)\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
>>>> line 847, in start\n message, payload = await self._protocol.read() # type:
>>>> ignore # noqa\n File
>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
>>>> line 591, in read\n await self._waiter\n
>>>>
>>>> The same error happened on both remote
>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>>>
>>>> Please advise.
>>>>
>>>> Thanks
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210618/dfc427dc/attachment.htm>
More information about the Pulp-list
mailing list