[Pulp-list] pulp 3.7.3 aarch64 repo
Bin Li (BLOOMBERG/ 120 PARK)
bli111 at bloomberg.net
Fri Jun 18 19:03:31 UTC 2021
Still have errors on this repo https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
Got 3 different errors for last 3 runs.
Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:17:22 pulpserver rq: rv = job.perform()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:17:22 pulpserver rq: dv.create()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:17:22 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:17:22 pulpserver rq: return future.result()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:17:22 pulpserver rq: await self.run()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:17:22 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:17:22 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 89, in _run
Jun 18 14:17:22 pulpserver rq: to_return = await self._handle_response(response)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", line 180, in _handle_response
Jun 18 14:17:22 pulpserver rq: chunk = await response.content.read(1048576) # 1 megabyte
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 368, in read
Jun 18 14:17:22 pulpserver rq: await self._wait('read')
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 296, in _wait
Jun 18 14:17:22 pulpserver rq: await waiter
Jun 18 14:17:22 pulpserver rq: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last):
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:17:22 pulpserver rq: rv = job.perform()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:17:22 pulpserver rq: self._result = self._execute()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:17:22 pulpserver rq: dv.create()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:17:22 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:17:22 pulpserver rq: return future.result()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:17:22 pulpserver rq: await self.run()
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:17:22 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:17:22 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 89, in _run
Jun 18 14:17:22 pulpserver rq: to_return = await self._handle_response(response)
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", line 180, in _handle_response
Jun 18 14:17:22 pulpserver rq: chunk = await response.content.read(1048576) # 1 megabyte
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 368, in read
Jun 18 14:17:22 pulpserver rq: await self._wait('read')
Jun 18 14:17:22 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 296, in _wait
Jun 18 14:17:22 pulpserver rq: await waiter
Jun 18 14:17:22 pulpserver rq: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
==================================
Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
Jun 18 14:34:09 pulpserver rq: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in create_connection
Jun 18 14:34:09 pulpserver rq: ssl_handshake_timeout=ssl_handshake_timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in _create_connection_transport
Jun 18 14:34:09 pulpserver rq: await waiter
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in data_received
Jun 18 14:34:09 pulpserver rq: ssldata, appdata = self._sslpipe.feed_ssldata(data)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause of the following exception:
Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:34:09 pulpserver rq: rv = job.perform()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:34:09 pulpserver rq: dv.create()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:34:09 pulpserver rq: return future.result()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:34:09 pulpserver rq: await self.run()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:34:09 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:34:09 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 87, in _run
Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 483, in _request
Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 523, in connect
Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req, traces, timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 856, in _create_connection
Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 1097, in _create_proxy_connection
Jun 18 14:34:09 pulpserver rq: req=req)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
Jun 18 14:34:09 pulpserver rq: raise ClientConnectorSSLError(req.connection_key, exc) from exc
Jun 18 14:34:09 pulpserver rq: aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)]
Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
Jun 18 14:34:09 pulpserver rq: return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in create_connection
Jun 18 14:34:09 pulpserver rq: ssl_handshake_timeout=ssl_handshake_timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in _create_connection_transport
Jun 18 14:34:09 pulpserver rq: await waiter
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in data_received
Jun 18 14:34:09 pulpserver rq: ssldata, appdata = self._sslpipe.feed_ssldata(data)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake
Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake()
Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)
Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause of the following exception:
Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last):
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:34:09 pulpserver rq: rv = job.perform()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:34:09 pulpserver rq: self._result = self._execute()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:34:09 pulpserver rq: dv.create()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:34:09 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:34:09 pulpserver rq: return future.result()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:34:09 pulpserver rq: await self.run()
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:34:09 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:34:09 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 87, in _run
Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 483, in _request
Jun 18 14:34:09 pulpserver rq: timeout=real_timeout
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 523, in connect
Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req, traces, timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 856, in _create_connection
Jun 18 14:34:09 pulpserver rq: req, traces, timeout)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 1097, in _create_proxy_connection
Jun 18 14:34:09 pulpserver rq: req=req)
Jun 18 14:34:09 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
Jun 18 14:34:09 pulpserver rq: raise ClientConnectorSSLError(req.connection_key, exc) from exc
Jun 18 14:34:09 pulpserver rq: aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056)]
=================================
Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ServerDisconnectedError
Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:47:28 pulpserver rq: rv = job.perform()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:47:28 pulpserver rq: dv.create()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:47:28 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:47:28 pulpserver rq: return future.result()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:47:28 pulpserver rq: await self.run()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:47:28 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:47:28 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 87, in _run
Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 504, in _request
Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start
Jun 18 14:47:28 pulpserver rq: message, payload = await self._protocol.read() # type: ignore # noqa
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 591, in read
Jun 18 14:47:28 pulpserver rq: await self._waiter
Jun 18 14:47:28 pulpserver rq: aiohttp.client_exceptions.ServerDisconnectedError
Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last):
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job
Jun 18 14:47:28 pulpserver rq: rv = job.perform()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform
Jun 18 14:47:28 pulpserver rq: self._result = self._execute()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute
Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize
Jun 18 14:47:28 pulpserver rq: dv.create()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create
Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline)
Jun 18 14:47:28 pulpserver rq: File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete
Jun 18 14:47:28 pulpserver rq: return future.result()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jun 18 14:47:28 pulpserver rq: await self.run()
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 152, in run
Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 178, in _handle_content_unit
Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*downloaders_for_content)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", line 88, in download
Jun 18 14:47:28 pulpserver rq: download_result = await downloader.run(extra_data=self.extra_data)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", line 227, in run
Jun 18 14:47:28 pulpserver rq: return await self._run(extra_data=extra_data)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", line 87, in _run
Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", line 504, in _request
Jun 18 14:47:28 pulpserver rq: await resp.start(conn)
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start
Jun 18 14:47:28 pulpserver rq: message, payload = await self._protocol.read() # type: ignore # noqa
Jun 18 14:47:28 pulpserver rq: File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", line 591, in read
Jun 18 14:47:28 pulpserver rq: await self._waiter
Jun 18 14:47:28 pulpserver rq: aiohttp.client_exceptions.ServerDisconnectedError
The aarch64 baseos repo completed successfully.
From: dalley at redhat.com At: 06/18/21 10:55:56 UTC-4:00To: dkliban at redhat.com
Cc: Bin Li (BLOOMBERG/ 120 PARK ) , pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo
replace "but only with the errors" with "but only with the normal types of errors"
On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dalley at redhat.com> wrote:
@Bin Li
Could you post the error that was logged? I think this traceback is missing some of the relevant details.
I can tell you what I expect that problem *might* be, because we've been hearing about it a lot recently. Again, this is just a guess, without more information.
We have found that some content delivery networks (including the Red Hat one) start returning seemingly irrelevant error messages when they get 'angry' about too much traffic coming from one source. Rather than returning HTTP 429 or some other standard response, sometimes it will just terminate the connection, or give you SSL errors, or return an incomplete payload. This might be some type of DDoS prevention / flood control mechanism, I'm not entirely sure. The mitigation is to either reduce the "download_concurrency" on your remote from 10 to 5, or using "on_demand" syncs rather than "immediate". Both of those are just to lessen the load on the CDN and make it less likely to start dropping connections.
But the "proper" fix is being worked on for pulpcore 3.14. We want to automatically retry downloads that fail using a backoff algorithm to prevent the entire sync from failing. We were already doing this, but only with the errors, rather than the weird ones that seem to come up in practice. And we're making the number of retry attempts configurable.
On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com> wrote:
I don't see anything special about these repos. Are you still experiencing this problem?
On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Tried to download aarch64 repo and got the errors below:
"description": "",
"traceback": " File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 266, in synchronize\n dv.create()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 152, in run\n pb.done += task.result() # download_count\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 178, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\", line 88, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\", line 227, in run\n return await self._run(extra_data=extra_data)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\", line 87, in _run\n async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response:\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", line 1012, in __aenter__\n self._resp = await self._coro\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", line 504, in _request\n await resp.start(conn)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\", line 847, in start\n message, payload = await self._protocol.read() # type: ignore # noqa\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\", line 591, in read\n await self._waiter\n
The same error happened on both remote https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
Please advise.
Thanks_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210618/72e73c31/attachment.htm>
More information about the Pulp-list
mailing list