[Pulp-list] pulp 3.7.3 aarch64 repo

Daniel Alley dalley at redhat.com
Fri Jun 18 14:52:17 UTC 2021


@Bin Li <bli111 at bloomberg.net>
Could you post the error that was logged?  I think this traceback is
missing some of the relevant details.

I can tell you what I expect that problem *might* be, because we've been
hearing about it a lot recently.  Again, this is just a guess, without more
information.

We have found that some content delivery networks (including the Red Hat
one) start returning seemingly irrelevant error messages when they get
'angry' about too much traffic coming from one source.  Rather than
returning HTTP 429 or some other standard response, sometimes it will just
terminate the connection, or give you SSL errors, or return an incomplete
payload.  This might be some type of DDoS prevention / flood control
mechanism, I'm not entirely sure.  The mitigation is to either reduce the
"download_concurrency" on your remote from 10 to 5, or using "on_demand"
syncs rather than "immediate". Both of those are just to lessen the load on
the CDN and make it less likely to start dropping connections.

But the "proper" fix is being worked on for pulpcore 3.14.  We want to
automatically retry downloads that fail using a backoff algorithm to
prevent the entire sync from failing.  We were already doing this, but only
with the errors, rather than the weird ones that seem to come up in
practice.  And we're making the number of retry attempts configurable.

On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com> wrote:

> I don't see anything special about these repos. Are you still experiencing
> this problem?
>
>
> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>> Tried to download aarch64 repo and got the errors below:
>>
>> "description": "",
>> "traceback": " File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
>> line 886, in perform_job\n rv = job.perform()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>> line 664, in perform\n self._result = self._execute()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>> line 266, in synchronize\n dv.create()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>> line 148, in create\n loop.run_until_complete(pipeline)\n File
>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
>> run_until_complete\n return future.result()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>> line 43, in __call__\n await self.run()\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>> line 152, in run\n pb.done += task.result() # download_count\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>> line 178, in _handle_content_unit\n await
>> asyncio.gather(*downloaders_for_content)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
>> line 88, in download\n download_result = await
>> downloader.run(extra_data=self.extra_data)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
>> line 227, in run\n return await self._run(extra_data=extra_data)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
>> auth=self.auth) as response:\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>> line 1012, in __aenter__\n self._resp = await self._coro\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>> line 504, in _request\n await resp.start(conn)\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
>> line 847, in start\n message, payload = await self._protocol.read() # type:
>> ignore # noqa\n File
>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
>> line 591, in read\n await self._waiter\n
>>
>> The same error happened on both remote
>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>
>> Please advise.
>>
>> Thanks
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://listman.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210618/8f910d02/attachment.htm>


More information about the Pulp-list mailing list