[Pulp-list] pulp 3.7.3 aarch64 repo

Ina Panova ipanova at redhat.com
Mon Jun 21 09:50:30 UTC 2021


+1 to what @Daniel Alley <dalley at redhat.com> said.
I'd encourage to use temporary workarounds mentioned by Daniel to mitigate
the issue and once 3.14 is out to upgrade if possible.

--------
Regards,

Ina Panova
Senior Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."


On Fri, Jun 18, 2021 at 9:55 PM Daniel Alley <dalley at redhat.com> wrote:

> Well.. the specific fix we need here *might* be backportable, although
> the full patch is not.  I can look into it next week, if the issues
> continue.
>
> On Fri, Jun 18, 2021 at 3:46 PM Daniel Alley <dalley at redhat.com> wrote:
>
>> Yup, that's the issue.  I've seen all 3 of those errors before while I
>> was testing my fix for this issue (retriable downloads).  The
>> cdn.redhat.com seems to be having a very bad day, today.
>>
>> It's a feature though, and not backportable, because the changes touch
>> the database and several different areas of code.  My only suggestions are
>> to either reduce the download_concurrency further, use on_demand sync so
>> that it doesn't immediately download packages at all, or wait (because the
>> CDN issue is likely temporary).  And upgrade once you get the chance (once
>> Katello packages pulpcore 3.14).
>>
>> On Fri, Jun 18, 2021 at 3:03 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> Still have errors on this repo
>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>>
>>>
>>> Got 3 different errors for last 3 runs.
>>> Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR:
>>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>>> completed
>>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:17:22 pulpserver rq: dv.create()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:17:22 pulpserver rq: return future.result()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:17:22 pulpserver rq: await self.run()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:17:22 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:17:22 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:17:22 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 89, in _run
>>> Jun 18 14:17:22 pulpserver rq: to_return = await
>>> self._handle_response(response)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
>>> line 180, in _handle_response
>>> Jun 18 14:17:22 pulpserver rq: chunk = await
>>> response.content.read(1048576) # 1 megabyte
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 368, in read
>>> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 296, in _wait
>>> Jun 18 14:17:22 pulpserver rq: await waiter
>>> Jun 18 14:17:22 pulpserver rq:
>>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>>> completed
>>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:17:22 pulpserver rq: rv = job.perform()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:17:22 pulpserver rq: dv.create()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:17:22 pulpserver rq: return future.result()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:17:22 pulpserver rq: await self.run()
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:17:22 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:17:22 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:17:22 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 89, in _run
>>> Jun 18 14:17:22 pulpserver rq: to_return = await
>>> self._handle_response(response)
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py",
>>> line 180, in _handle_response
>>> Jun 18 14:17:22 pulpserver rq: chunk = await
>>> response.content.read(1048576) # 1 megabyte
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 368, in read
>>> Jun 18 14:17:22 pulpserver rq: await self._wait('read')
>>> Jun 18 14:17:22 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 296, in _wait
>>> Jun 18 14:17:22 pulpserver rq: await waiter
>>> Jun 18 14:17:22 pulpserver rq:
>>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not
>>> completed
>>> ==================================
>>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 936, in _wrap_create_connection
>>> Jun 18 14:34:09 pulpserver rq: return await
>>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
>>> create_connection
>>> Jun 18 14:34:09 pulpserver rq:
>>> ssl_handshake_timeout=ssl_handshake_timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
>>> _create_connection_transport
>>> Jun 18 14:34:09 pulpserver rq: await waiter
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
>>> data_received
>>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
>>> self._sslpipe.feed_ssldata(data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
>>> feed_ssldata
>>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
>>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
>>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
>>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause
>>> of the following exception:
>>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:34:09 pulpserver rq: dv.create()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:34:09 pulpserver rq: return future.result()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:34:09 pulpserver rq: await self.run()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:34:09 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:34:09 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:34:09 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 87, in _run
>>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
>>> proxy=self.proxy, auth=self.auth) as response:
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 1012, in __aenter__
>>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 483, in _request
>>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 523, in connect
>>> Jun 18 14:34:09 pulpserver rq: proto = await
>>> self._create_connection(req, traces, timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 856, in _create_connection
>>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 1097, in _create_proxy_connection
>>> Jun 18 14:34:09 pulpserver rq: req=req)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 941, in _wrap_create_connection
>>> Jun 18 14:34:09 pulpserver rq: raise
>>> ClientConnectorSSLError(req.connection_key, exc) from exc
>>> Jun 18 14:34:09 pulpserver rq:
>>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
>>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
>>> sslv3 alert unexpected message (_ssl.c:1056)]
>>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 936, in _wrap_create_connection
>>> Jun 18 14:34:09 pulpserver rq: return await
>>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in
>>> create_connection
>>> Jun 18 14:34:09 pulpserver rq:
>>> ssl_handshake_timeout=ssl_handshake_timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in
>>> _create_connection_transport
>>> Jun 18 14:34:09 pulpserver rq: await waiter
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in
>>> data_received
>>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata =
>>> self._sslpipe.feed_ssldata(data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in
>>> feed_ssldata
>>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
>>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
>>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL:
>>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
>>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause
>>> of the following exception:
>>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:34:09 pulpserver rq: rv = job.perform()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:34:09 pulpserver rq: dv.create()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:34:09 pulpserver rq: return future.result()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:34:09 pulpserver rq: await self.run()
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:34:09 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:34:09 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:34:09 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 87, in _run
>>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url,
>>> proxy=self.proxy, auth=self.auth) as response:
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 1012, in __aenter__
>>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 483, in _request
>>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 523, in connect
>>> Jun 18 14:34:09 pulpserver rq: proto = await
>>> self._create_connection(req, traces, timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 856, in _create_connection
>>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 1097, in _create_proxy_connection
>>> Jun 18 14:34:09 pulpserver rq: req=req)
>>> Jun 18 14:34:09 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py",
>>> line 941, in _wrap_create_connection
>>> Jun 18 14:34:09 pulpserver rq: raise
>>> ClientConnectorSSLError(req.connection_key, exc) from exc
>>> Jun 18 14:34:09 pulpserver rq:
>>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host
>>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE]
>>> sslv3 alert unexpected message (_ssl.c:1056)]
>>>
>>> =================================
>>> Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR:
>>> aiohttp.client_exceptions.ServerDisconnectedError
>>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:47:28 pulpserver rq: dv.create()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:47:28 pulpserver rq: return future.result()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:47:28 pulpserver rq: await self.run()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:47:28 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:47:28 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:47:28 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 87, in _run
>>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
>>> proxy=self.proxy, auth=self.auth) as response:
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 1012, in __aenter__
>>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 504, in _request
>>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
>>> line 847, in start
>>> Jun 18 14:47:28 pulpserver rq: message, payload = await
>>> self._protocol.read() # type: ignore # noqa
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 591, in read
>>> Jun 18 14:47:28 pulpserver rq: await self._waiter
>>> Jun 18 14:47:28 pulpserver rq:
>>> aiohttp.client_exceptions.ServerDisconnectedError
>>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py",
>>> line 886, in perform_job
>>> Jun 18 14:47:28 pulpserver rq: rv = job.perform()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 664, in perform
>>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line
>>> 670, in _execute
>>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args,
>>> **self.kwargs)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py",
>>> line 266, in synchronize
>>> Jun 18 14:47:28 pulpserver rq: dv.create()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py",
>>> line 148, in create
>>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in
>>> run_until_complete
>>> Jun 18 14:47:28 pulpserver rq: return future.result()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 225, in create_pipeline
>>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py",
>>> line 43, in __call__
>>> Jun 18 14:47:28 pulpserver rq: await self.run()
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 152, in run
>>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py",
>>> line 178, in _handle_content_unit
>>> Jun 18 14:47:28 pulpserver rq: await
>>> asyncio.gather(*downloaders_for_content)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py",
>>> line 88, in download
>>> Jun 18 14:47:28 pulpserver rq: download_result = await
>>> downloader.run(extra_data=self.extra_data)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py",
>>> line 227, in run
>>> Jun 18 14:47:28 pulpserver rq: return await
>>> self._run(extra_data=extra_data)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py",
>>> line 87, in _run
>>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url,
>>> proxy=self.proxy, auth=self.auth) as response:
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 1012, in __aenter__
>>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py",
>>> line 504, in _request
>>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py",
>>> line 847, in start
>>> Jun 18 14:47:28 pulpserver rq: message, payload = await
>>> self._protocol.read() # type: ignore # noqa
>>> Jun 18 14:47:28 pulpserver rq: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py",
>>> line 591, in read
>>> Jun 18 14:47:28 pulpserver rq: await self._waiter
>>> Jun 18 14:47:28 pulpserver rq:
>>> aiohttp.client_exceptions.ServerDisconnectedError
>>>
>>>
>>> The aarch64 baseos repo completed successfully.
>>>
>>>
>>>
>>>
>>> From: dalley at redhat.com At: 06/18/21 10:55:56 UTC-4:00
>>> To: dkliban at redhat.com
>>> Cc: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>,
>>> pulp-list at redhat.com
>>> Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo
>>>
>>> replace "but only with the errors" with "but only with the normal types
>>> of errors"
>>>
>>> On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dalley at redhat.com> wrote:
>>>
>>>> @Bin Li <bli111 at bloomberg.net>
>>>> Could you post the error that was logged?  I think this traceback is
>>>> missing some of the relevant details.
>>>>
>>>> I can tell you what I expect that problem *might* be, because we've
>>>> been hearing about it a lot recently.  Again, this is just a guess, without
>>>> more information.
>>>>
>>>> We have found that some content delivery networks (including the Red
>>>> Hat one) start returning seemingly irrelevant error messages when they get
>>>> 'angry' about too much traffic coming from one source.  Rather than
>>>> returning HTTP 429 or some other standard response, sometimes it will just
>>>> terminate the connection, or give you SSL errors, or return an incomplete
>>>> payload.  This might be some type of DDoS prevention / flood control
>>>> mechanism, I'm not entirely sure.  The mitigation is to either reduce the
>>>> "download_concurrency" on your remote from 10 to 5, or using "on_demand"
>>>> syncs rather than "immediate". Both of those are just to lessen the load on
>>>> the CDN and make it less likely to start dropping connections.
>>>>
>>>> But the "proper" fix is being worked on for pulpcore 3.14.  We want to
>>>> automatically retry downloads that fail using a backoff algorithm to
>>>> prevent the entire sync from failing.  We were already doing this, but only
>>>> with the errors, rather than the weird ones that seem to come up in
>>>> practice.  And we're making the number of retry attempts configurable.
>>>>
>>>> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com>
>>>> wrote:
>>>>
>>>>> I don't see anything special about these repos. Are you still
>>>>> experiencing this problem?
>>>>>
>>>>>
>>>>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
>>>>> bli111 at bloomberg.net> wrote:
>>>>>
>>>>>> Tried to download aarch64 repo and got the errors below:
>>>>>>
>>>>>> "description": "",
>>>>>> "traceback": " File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
>>>>>> line 886, in perform_job\n rv = job.perform()\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>>>> line 664, in perform\n self._result = self._execute()\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>>>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>>>>>> line 266, in synchronize\n dv.create()\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>>>>>> line 148, in create\n loop.run_until_complete(pipeline)\n File
>>>>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
>>>>>> run_until_complete\n return future.result()\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>>>>> line 43, in __call__\n await self.run()\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>>>> line 152, in run\n pb.done += task.result() # download_count\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>>>>> line 178, in _handle_content_unit\n await
>>>>>> asyncio.gather(*downloaders_for_content)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
>>>>>> line 88, in download\n download_result = await
>>>>>> downloader.run(extra_data=self.extra_data)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
>>>>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
>>>>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
>>>>>> auth=self.auth) as response:\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>>>> line 1012, in __aenter__\n self._resp = await self._coro\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>>>>> line 504, in _request\n await resp.start(conn)\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
>>>>>> line 847, in start\n message, payload = await self._protocol.read() # type:
>>>>>> ignore # noqa\n File
>>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
>>>>>> line 591, in read\n await self._waiter\n
>>>>>>
>>>>>> The same error happened on both remote
>>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
>>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>>>>>
>>>>>> Please advise.
>>>>>>
>>>>>> Thanks
>>>>>> _______________________________________________
>>>>>> Pulp-list mailing list
>>>>>> Pulp-list at redhat.com
>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-list mailing list
>>>>> Pulp-list at redhat.com
>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>>
>>> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210621/121fb46d/attachment.htm>


More information about the Pulp-list mailing list