It looks like the last error caused failed sync process. I reset the db on a dev host and try to sync the same upstream repo and got
""An error occurred (QuotaExceeded) when calling the PutObject operation: Unknown"
Any idea how to fix this?
"traceback": " File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 266, in synchronize\n dv.create()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 219, in run\n d_artifact.artifact for d_artifact in da_to_save\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/content.py\", line 87, in bulk_get_or_create\n return super().bulk_create(objs, batch_size=batch_size)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 468, in bulk_create\n self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1204, in _batched_insert\n ignore_conflicts=ignore_conflicts,\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1186, in _insert\n return query.get_compiler(using=using).execute_sql(return_id)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1376, in execute_sql\n for sql, params in self.as_sql():\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django_readonly_field/compiler.py\", line 31, in as_sql\n return super(ReadonlySQLCompilerMixin, self).as_sql()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in as_sql\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in <listcomp>\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1319, in <listcomp>\n [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1270, in pre_save_val\n return field.pre_save(obj, add=True)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/fields.py\", line 68, in pre_save\n return super().pre_save(model_instance, add)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 288, in pre_save\n file.save(file.name, file.file, save=False)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 87, in save\n self.name = self.storage.save(name, content, max_length=self.field.max_length)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/files/storage.py\", line 52, in save\n return self._save(name, content)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/storages/backends/s3boto3.py\", line 447, in _save\n obj.upload_fileobj(content, ExtraArgs=params)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 621, in object_upload_fileobj\n ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 539, in upload_fileobj\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 106, in result\n return self._coordinator.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 265, in result\n raise self._exception\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 126, in __call__\n return self._execute_main(kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 150, in _execute_main\n return_value = self._main(**kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/upload.py\", line 692, in _main\n client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 357, in _api_call\n return self._make_api_call(operation_name, kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 676, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"
From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 11:34:57To: daviddavis at redhat.com
Cc: bmbouter at redhat.com, pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error
Got a new errors "Package matching query does not exist.". Is this also related to upstream repo? Can we have more details when this happens?
From: daviddavis at redhat.com At: 03/05/21 15:06:40To: Bin Li (BLOOMBERG/ 120 PARK )
Cc: bmbouter at redhat.com, pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error
Great, thanks for the update.
David
On Fri, Mar 5, 2021 at 2:47 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
Thanks Dave. Got the filename which has the inconsistent checksum after patching. We will ask upstream remote to update the repodata.
From: daviddavis at redhat.com At: 03/05/21 12:42:56To: bmbouter at redhat.com
Cc: Bin Li (BLOOMBERG/ 120 PARK ) , pulp-list at redhat.com
Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error
Regarding the error message, I've observed the problem myself. and have filed an issue:
https://pulp.plan.io/issues/8357
In the meantime, if you can patch the code, this should tell you give you more info:
https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31
David
On Fri, Mar 5, 2021 at 12:35 PM Brian Bouterse <bmbouter at redhat.com> wrote:
Did this happen inside a task? Did you see a traceback for it also?
On Fri, Mar 5, 2021 at 12:00 PM Bin Li (BLOOMBERG/ 120 PARK) <bli111 at bloomberg.net> wrote:
The sync process gave an error "A file failed validation due to checksum". Is this error caused by remote repo? Is there a way to find out which file cause the issue?_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com
https://listman.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20210309/f9849597/attachment.htm>