<html><head><title></title></head><body><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);">Please ignore the last message. It is irrelevant.<div>We actually able to sync the upstream repo successfully after we reset the database. The question is how to we recover from previous failure? We keep getting "Package matching query does not exist." without reset the database. Recreating the repo didn't help either.<br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><br><div class="bbg-rte-fold-content" data-header="From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 17:06:53"data-digest="From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 17:06:53" style=""><div class="bbg-rte-fold-summary">From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 17:06:53</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" href="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" href="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);">It looks like the last error caused failed sync process. I reset the db on a dev host and try to sync the same upstream repo and got <div>""An error occurred (QuotaExceeded) when calling the PutObject operation: Unknown"</div><div><br></div><div>Any idea how to fix this?</div><div><br></div><div><br></div><div> "traceback": " File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 266, in synchronize\n dv.create()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 219, in run\n d_artifact.artifact for d_artifact in da_to_save\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/content.py\", line 87, in bulk_get_or_create\n return super().bulk_create(objs, batch_size=batch_size)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 468, in bulk_create\n self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1204, in _batched_insert\n ignore_conflicts=ignore_conflicts,\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1186, in _insert\n return query.get_compiler(using=using).execute_sql(return_id)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1376, in execute_sql\n for sql, params in self.as_sql():\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django_readonly_field/compiler.py\", line 31, in as_sql\n return super(ReadonlySQLCompilerMixin, self).as_sql()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in as_sql\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in <listcomp>\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1319, in <listcomp>\n [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1270, in pre_save_val\n return field.pre_save(obj, add=True)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/fields.py\", line 68, in pre_save\n return super().pre_save(model_instance, add)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 288, in pre_save\n file.save(file.name, file.file, save=False)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 87, in save\n self.name = self.storage.save(name, content, max_length=self.field.max_length)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/files/storage.py\", line 52, in save\n return self._save(name, content)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/storages/backends/s3boto3.py\", line 447, in _save\n obj.upload_fileobj(content, ExtraArgs=params)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 621, in object_upload_fileobj\n ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 539, in upload_fileobj\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 106, in result\n return self._coordinator.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 265, in result\n raise self._exception\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 126, in __call__\n return self._execute_main(kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 150, in _execute_main\n return_value = self._main(**kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/upload.py\", line 692, in _main\n client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 357, in _api_call\n return self._make_api_call(operation_name, kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 676, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"</div><div><br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><br><div class="bbg-rte-fold-content" data-header="From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 11:34:57"data-digest="From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 11:34:57" style=""><div class="bbg-rte-fold-summary">From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 11:34:57</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" href="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" href="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);">Got a new errors "Package matching query does not exist.". Is this also related to upstream repo? Can we have more details when this happens?<br><br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><div><div class="bbg-rte-fold-content" data-header="From: daviddavis@redhat.com At: 03/05/21 15:06:40" data-digest="From: daviddavis@redhat.com At: 03/05/21 15:06:40" style=""><div class="bbg-rte-fold-summary">From: daviddavis@redhat.com At: 03/05/21 15:06:40</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" href="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" href="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div class="rte-internet-block-wrapper" style="background: white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div class="rte-internet-block"><blockquote><div dir="ltr">Great, thanks for the update. <br><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><br></div><div>David</div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 2:47 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">Thanks Dave. Got the filename which has the inconsistent checksum after patching. We will ask upstream remote to update the repodata.<br><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:rte:bind">daviddavis@redhat.com</a> At: 03/05/21 12:42:56</div>To: <a spellcheck="false" bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr">Regarding the error message, I've observed the problem myself. and have filed an issue:<div><br></div><div><a spellcheck="false" bbg-destination="rte:bind" class=""href="https://pulp.plan.io/issues/8357" data-destination="rte:bind">https://pulp.plan.io/issues/8357</a><br><div><br></div><div>In the meantime, if you can patch the code, this should tell you give you more info:</div><div><br></div><div><a spellcheck="false" bbg-destination="rte:bind" class=""href="https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31"data-destination="rte:bind">https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31</a><br></div><div><div><br></div><div>David</div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 12:35 PM Brian Bouterse <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:rte:bind">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Did this happen inside a task? Did you see a traceback for it also?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 12:00 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">The sync process gave an error "A file failed validation due to checksum". Is this error caused by remote repo? Is there a way to find out which file cause the issue?</div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> _______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></div></div></div></div></div></body></html>