[Pulp-list] pulp 3.7.3 aarch64 repo

[Daniel Alley]

replace "but only with the errors" with "but only with the normal types of
errors"

On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dalley at redhat.com> wrote:

> @Bin Li <bli111 at bloomberg.net>
> Could you post the error that was logged?  I think this traceback is
> missing some of the relevant details.
>
> I can tell you what I expect that problem *might* be, because we've been
> hearing about it a lot recently.  Again, this is just a guess, without more
> information.
>
> We have found that some content delivery networks (including the Red Hat
> one) start returning seemingly irrelevant error messages when they get
> 'angry' about too much traffic coming from one source.  Rather than
> returning HTTP 429 or some other standard response, sometimes it will just
> terminate the connection, or give you SSL errors, or return an incomplete
> payload.  This might be some type of DDoS prevention / flood control
> mechanism, I'm not entirely sure.  The mitigation is to either reduce the
> "download_concurrency" on your remote from 10 to 5, or using "on_demand"
> syncs rather than "immediate". Both of those are just to lessen the load on
> the CDN and make it less likely to start dropping connections.
>
> But the "proper" fix is being worked on for pulpcore 3.14.  We want to
> automatically retry downloads that fail using a backoff algorithm to
> prevent the entire sync from failing.  We were already doing this, but only
> with the errors, rather than the weird ones that seem to come up in
> practice.  And we're making the number of retry attempts configurable.
>
> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkliban at redhat.com> wrote:
>
>> I don't see anything special about these repos. Are you still
>> experiencing this problem?
>>
>>
>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> Tried to download aarch64 repo and got the errors below:
>>>
>>> "description": "",
>>> "traceback": " File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\",
>>> line 886, in perform_job\n rv = job.perform()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>> line 664, in perform\n self._result = self._execute()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\",
>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
>>> line 266, in synchronize\n dv.create()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\",
>>> line 148, in create\n loop.run_until_complete(pipeline)\n File
>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in
>>> run_until_complete\n return future.result()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\",
>>> line 43, in __call__\n await self.run()\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>> line 152, in run\n pb.done += task.result() # download_count\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
>>> line 178, in _handle_content_unit\n await
>>> asyncio.gather(*downloaders_for_content)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\",
>>> line 88, in download\n download_result = await
>>> downloader.run(extra_data=self.extra_data)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\",
>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\",
>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy,
>>> auth=self.auth) as response:\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>> line 1012, in __aenter__\n self._resp = await self._coro\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\",
>>> line 504, in _request\n await resp.start(conn)\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\",
>>> line 847, in start\n message, payload = await self._protocol.read() # type:
>>> ignore # noqa\n File
>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\",
>>> line 591, in read\n await self._waiter\n
>>>
>>> The same error happened on both remote
>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and
>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os
>>>
>>> Please advise.
>>>
>>> Thanks
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://listman.redhat.com/mailman/listinfo/pulp-list
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://listman.redhat.com/mailman/listinfo/pulp-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210618/bf2fff4a/attachment.htm>