<html><head><title></title></head><body><div class="rte-style-maintainer rte-pre-wrap" data-color="global-default" bbg-color="default" data-bb-font-size="medium"bbg-font-size="medium" style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><br>The core_taskreservedresource is actually empty after restoring the database. I found 2 tasks in waiting state from 10 month ago. I got a 409 conflict error when try to delete them with api. I then deleted both tasks from core_task table. The sync job was able to run but got the same "Package matching query does not exist". It looks like the issue existed before we tried to sync a week ago. I am not sure what else we can do except a reset of the database from scratch. We don't add any packages locally. All packages are synced from the upstream. The only concern is how to keep distributions with the same content. Some of our distributions point to older version of repo. Is there a way to make these distribution to have the same content after reset database?<div><br></div><div><br><div><br><div class="rte-style-maintainer" data-color="global-default"bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; color: rgb(0, 0, 0);"><div><div class="bbg-rte-fold-content" data-header="From: dalley@redhat.com At: 03/19/21 11:08:32" data-digest="From: dalley@redhat.com At: 03/19/21 11:08:32" style=""><div class="bbg-rte-fold-summary">From: dalley@redhat.com At: 03/19/21 11:08:32</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" href="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:daviddavis@redhat.com" href="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" href="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div class="rte-internet-block-wrapper" style="background: white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div class="rte-internet-block"><blockquote><div dir="ltr"><div>I believe the correct procedure is to stop the workers, delete all of the items in the "reserved_resources" table, and restart the workers.<br></div><div><br></div><div>You are still using Pulpcore 3.7, correct? Once this is sorted, I would recommend trying to upgrade... which I'm sure doesn't sound appealing right now... but a lot of work was done in the 3.9 - 3.10 timeframe to prevent tasking system deadlocks like this, as well as some improvements around being able to check for corrupted artifacts. <br></div><div><br></div><div><a spellcheck="false" bbg-destination="rte:bind" class=""href="https://yum.theforeman.org/pulpcore/3.9/" data-destination="rte:bind">https://yum.theforeman.org/pulpcore/3.9/</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 19, 2021 at 9:25 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">I tried to restore the database on a dev instance. I was able to recreate repo and remote with the restored database but the sync stucked on waiting. No other jobs was running. The dev instance has a different hostname. How do we find out why the sync job is in the waiting state?<div><br></div><div><br><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:rte:bind">daviddavis@redhat.com</a> At: 03/18/21 15:30:29</div>To: <a spellcheck="false" bbg-destination="mailto:dalley@redhat.com" class="" href="mailto:dalley@redhat.com"data-destination="mailto:dalley@redhat.com"> dalley@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr">I'm guessing there's a package that got saved with a bad pkgId. I think what dalley recommends should work. I'd also make a backup of your current database in case rolling back causes bigger problems.<br><div><div><br></div><div>David</div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 18, 2021 at 2:32 PM Daniel Alley <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Restoring postgresql from tape should fix your database, yes.</div><div><br></div><div>I'm not 100% sure what happens if the artifact store has untracked files, or missing files that the database expects to be there. In newer versions of pulpcore we have a "repair" feature to help deal with such issues, but 3.7 predates it.<br></div><div> <br></div><div>I would say, take a backup of your /var/lib/pulp directory and then try the database restore.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 18, 2021 at 10:23 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;"><br>We still need to restore our instance to the previous state before this happened. We already get the postgresql dump from tape. Will restoring database work in this case?<div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a> At: 03/17/21 14:45:15</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr">Thanks!<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 17, 2021 at 11:34 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;"><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">FYI, I filed a new issue <a spellcheck="false"bbg-destination="rte:bind" class="" href="https://pulp.plan.io/issues/8411" data-destination="rte:bind">https://pulp.plan.io/issues/8411</a> to track this.</div><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/16/21 15:31:14</div>To: <a spellcheck="false" bbg-destination="mailto:dalley@redhat.com" class="" href="mailto:dalley@redhat.com"data-destination="mailto:dalley@redhat.com"> dalley@redhat.com</a>, <a spellcheck="false" bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div>I checked the metadate from primary.xml. The sha1sum matches with actual file.</div><div><br></div><metadata xmlns="<a spellcheck="false" bbg-destination="rte:bind" class="" href="http://linux.duke.edu/metadata/common"data-destination="rte:bind">http://linux.duke.edu/metadata/common</a>" xmlns:rpm="<a spellcheck="false"bbg-destination="rte:bind" class="" href="http://linux.duke.edu/metadata/rpm" data-destination="rte:bind">http://linux.duke.edu/metadata/rpm</a>"><div> <package type="rpm"></div><div> <name>flume</name></div><div> <arch>noarch</arch></div><div> <version epoch="0" ver="1.9.0" rel="1"/></div><div> <checksum type="sha" pkgid="YES">b8b257c32135daf51e703d439594f1a676871d7d</checksum></div><div><br></div><div> # sha1sum flume-1.9.0-1.noarch.rpm</div><div>b8b257c32135daf51e703d439594f1a676871d7d flume-1.9.0-1.noarch.rpm</div><div><br></div><div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/16/21 15:06:43</div>To: <a spellcheck="false" bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:dalley@redhat.com" class="" href="mailto:dalley@redhat.com" data-destination="mailto:dalley@redhat.com"> dalley@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">I downloaded the package and I didn't find the check_sum in rpm_checksum.<div><br></div><div># sha256sum flume-1.9.0-1.noarch.rpm</div><div>c7fcec6d3385c079af6ed83cb272f52cbe2cb30ca93ed91732b1c8698b2ad76b flume-1.9.0-1.noarch.rpm</div><div><br></div><div>=> select checksum from rpm_checksum where checksum like '%a93ed91732b1c8698b2ad76b'; </div><div><br></div><div> checksum </div><div>----------</div><div>(0 rows)</div><div><br><div>Also,there is no result from core_contentartifact</div><div><br></div>=> select * from core_contentartifact where relative_path like '%flume%';<div> pulp_id | pulp_created | pulp_last_updated | relative_path | artifact_id | content_id </div><div>---------+--------------+-------------------+---------------+-------------+------------</div><div>(0 rows)</div><div><br></div><div><br></div><div>The upstream repo was fixed. I had no issues when I syncd from a fresh empty pulp instance.</div><div><br></div><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a> At: 03/16/21 14:37:29</div>To: <a spellcheck="false" bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr"><div>Also, the checksum of the package is stored as "pkgId" rather than as "checksum". Simply because that's what the RPM tools call it. There's no field actually named "checksum".<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>pulp=> select * from rpm_package;<br> content_ptr_id | name | epoch | version | release | arch | pkgId | checksum_type | summary | description | url | changelogs | files | requires | provides | conflicts | obsoletes | suggests | enhances | recommends | supplements | location<br>_base | location_href | rpm_buildhost | rpm_group | rpm_license | rpm_packager | rpm_sourcerpm | rpm_vendor | rpm_header_start | rpm_header_end | is_modular | size_archive | size_installed | size_package | time_build | time_file | evr <br>----------------+------+-------+---------+---------+------+-------+---------------+---------+-------------+-----+------------+-------+----------+----------+-----------+-----------+----------+----------+------------+-------------+---------<br>------+---------------+---------------+-----------+-------------+--------------+---------------+------------+------------------+----------------+------------+--------------+----------------+--------------+------------+-----------+-----<br>(0 rows)<br></div></blockquote><div><br></div><div>Going back to the original error though, this almost sounds like the file being downloaded doesn't match the checksum it's supposed to have. <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Received checksum b8b257c32135daf51e703d439594f1a676871d7d for <a spellcheck="false"bbg-destination="rte:bind" class="" href="http://something/something/flume-1.9.0-1.noarch.rpm"data-destination="rte:bind">http://something/something/flume-1.9.0-1.noarch.rpm</a> but expected c281a94a354178c42800d47b63479c2621772351</div></blockquote><div><br></div><div>Is there any chance you could manually download that file and verify its checksum, to make sure it's not corrupted at the source?<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 16, 2021 at 2:24 PM Daniel Alley <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I tried to read content from v3/content/. There is too much content to be listed. Not sure if I can specify a regex so I use select from db directly </div></blockquote><div><br></div><div>You can use query parameters when making HTTP calls against Pulp, like so:</div><div><br></div><div>POST .../pulp/api/v3/content/packages/ name=flume</div><div><br></div><div>There's a bunch of options available, they are documented here:<a spellcheck="false"bbg-destination="rte:bind" class="" href="https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/content_rpm_packages_list"data-destination="rte:bind">https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/content_rpm_packages_list</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 16, 2021 at 12:44 PM Brian Bouterse <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:rte:bind">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">This doesn't help you today, but I think this type of use case is what motivates an API call like this one that is being discussed: <a spellcheck="false"bbg-destination="rte:bind" class="" href="https://pulp.plan.io/issues/8372" data-destination="rte:bind">https://pulp.plan.io/issues/8372</a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 16, 2021 at 12:30 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">I tried to read content from v3/content/. There is too much content to be listed. Not sure if I can specify a regex so I use select from db directly to see if I can find the package which causes the issue originally. The query returns 0 rows. It looks like it was cleaned out unless another content causes this issue. Let me know if there is anything else I can try.<div><br></div><div>Below is what I have tried.</div><div><br></div><div>The original error:</div><div>Received checksum b8b257c32135daf51e703d439594f1a676871d7d for <a spellcheck="false"bbg-destination="rte:bind" class="" href="http://something/something/flume-1.9.0-1.noarch.rpm"data-destination="rte:bind">http://something/something/flume-1.9.0-1.noarch.rpm</a> but expected c281a94a354178c42800d47b63479c2621772351</div><div><br></div><div>=> select name from rpm_package where name like 'flume%' limit 100;</div><div> name </div><div>------</div><div>(0 rows)</div><div><br></div><div>=> select checksum from rpm_checksum where checksum like '%594f1a676871d7d' OR checksum like '%63479c2621772351';</div><div> checksum </div><div>----------</div><div>(0 rows)</div><div><br></div><div><br></div><div><br></div><div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a> At: 03/15/21 11:03:48</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr"><div>Do you know if that package could possibly have been present in any other repositories also? If you know which content unit it is, try accessing it via the content API after having done the orphan cleanup. If it still exists, it wasn't cleaned up for some reason, which may mean it's used by some other repository.<br></div><div><br></div><div>This is... interesting. Pulp seems to be attempting to save the package, hitting an IntegrityError because it already exists (expected), and then trying to retrieve the package, and not being able to find it.</div><div><br></div><div>Please file an issue with all the information you've posted so far, we will look into how this could be happening.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 15, 2021 at 9:37 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">I deleted the repo which failed sync and run "delete localhost/pulp/api/v3/orphans/" but I am still getting the same messages.<div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/12/21 12:18:44</div>To: <a spellcheck="false" bbg-destination="mailto:dalley@redhat.com" class="" href="mailto:dalley@redhat.com"data-destination="mailto:dalley@redhat.com"> dalley@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">Hi Dan,<div><br></div><div>Here is the traceback?</div><div><br></div><div> "error": {</div><div> "description": "Package matching query does not exist.", </div><div> "traceback": " File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 266, in synchronize\n dv.create()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/content_stages.py\", line 105, in run\n d_content.content.q()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 408, in get\n self.model._meta.object_name\n"</div><div><br></div><div><br></div><div>I will try removing the repo first and deleting orphans.</div><div><br></div><div><br><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:dalley@redhat.com" data-destination="mailto:rte:bind">dalley@redhat.com</a> At: 03/12/21 11:19:17</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr"><div>Hi Bin,</div><div><br></div><div>It's difficult to understand what exactly this error is. Is it an error message being printed out nicely, or part of a Python exception bubbling up? And if it's the latter, do you have the rest of the traceback?</div><div><br></div><div>You can't manually delete specific content units but you can delete "orphan" content units that aren't part of any repository. So if you know the content unit in question, you can delete it from your repositories, and then run orphan cleanup.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 11, 2021 at 11:27 AM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">If inconsistent repo data can cause Pulp to become unrecoverable, this is very difficult to prevent. Any inconsistent updates from upstream could potentially cause this to happen. At this point, we are thinking to restore the database from the backup before this sync issue happened, will this approach work? </div><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br></div><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/10/21 14:41:38</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><br></div>We know the rpm name which has different check sum with repodata and caused the sync failure earlier. I am guessing the current issue is caused by this rpm. Is there any way we can remove it from database? <div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 17:47:17</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">Please ignore the last message. It is irrelevant.<div>We actually able to sync the upstream repo successfully after we reset the database. The question is how to we recover from previous failure? We keep getting "Package matching query does not exist." without reset the database. Recreating the repo didn't help either.<br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 17:06:53</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">It looks like the last error caused failed sync process. I reset the db on a dev host and try to sync the same upstream repo and got <div>""An error occurred (QuotaExceeded) when calling the PutObject operation: Unknown"</div><div><br></div><div>Any idea how to fix this?</div><div><br></div><div><br></div><div> "traceback": " File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 266, in synchronize\n dv.create()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in run_until_complete\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 219, in run\n d_artifact.artifact for d_artifact in da_to_save\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/content.py\", line 87, in bulk_get_or_create\n return super().bulk_create(objs, batch_size=batch_size)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 468, in bulk_create\n self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1204, in _batched_insert\n ignore_conflicts=ignore_conflicts,\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py\", line 1186, in _insert\n return query.get_compiler(using=using).execute_sql(return_id)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1376, in execute_sql\n for sql, params in self.as_sql():\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django_readonly_field/compiler.py\", line 31, in as_sql\n return super(ReadonlySQLCompilerMixin, self).as_sql()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in as_sql\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1320, in <listcomp>\n for obj in self.query.objs\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1319, in <listcomp>\n [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1270, in pre_save_val\n return field.pre_save(obj, add=True)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/fields.py\", line 68, in pre_save\n return super().pre_save(model_instance, add)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 288, in pre_save\n file.save(<a spellcheck="false"bbg-destination="rte:bind" class="" href="file.name"data-destination="rte:bind">file.name</a>, file.file, save=False)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/fields/files.py\", line 87, in save\n <a spellcheck="false"bbg-destination="rte:bind" class="" href="self.name"data-destination="rte:bind">self.name</a> = self.storage.save(name, content, max_length=self.field.max_length)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/files/storage.py\", line 52, in save\n return self._save(name, content)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/storages/backends/s3boto3.py\", line 447, in _save\n obj.upload_fileobj(content, ExtraArgs=params)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 621, in object_upload_fileobj\n ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/boto3/s3/inject.py\", line 539, in upload_fileobj\n return future.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 106, in result\n return self._coordinator.result()\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/futures.py\", line 265, in result\n raise self._exception\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 126, in __call__\n return self._execute_main(kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/tasks.py\", line 150, in _execute_main\n return_value = self._main(**kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/s3transfer/upload.py\", line 692, in _main\n client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 357, in _api_call\n return self._make_api_call(operation_name, kwargs)\n File \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/botocore/client.py\", line 676, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"</div><div><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><br><div><div>From: Bin Li (BLOOMBERG/ 120 PARK) At: 03/09/21 11:34:57</div>To: <a spellcheck="false" bbg-destination="mailto:daviddavis@redhat.com" class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:daviddavis@redhat.com"> daviddavis@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br><div style="white-space: pre-wrap; font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";">Got a new errors "Package matching query does not exist.". Is this also related to upstream repo? Can we have more details when this happens?<br><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:rte:bind">daviddavis@redhat.com</a> At: 03/05/21 15:06:40</div>To: <a spellcheck="false" bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr">Great, thanks for the update. <br><div><div><br></div><div>David</div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 2:47 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">Thanks Dave. Got the filename which has the inconsistent checksum after patching. We will ask upstream remote to update the repodata.<br><br><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth";"><div><div><div>From: <a spellcheck="false" bbg-destination="mailto:rte:bind"class="" href="mailto:daviddavis@redhat.com" data-destination="mailto:rte:bind">daviddavis@redhat.com</a> At: 03/05/21 12:42:56</div>To: <a spellcheck="false" bbg-destination="mailto:bmbouter@redhat.com" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:bmbouter@redhat.com"> bmbouter@redhat.com</a><br>Cc: <a spellcheck="false"bbg-destination="mailto:bli111@bloomberg.net" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:bli111@bloomberg.net"> Bin Li (BLOOMBERG/ 120 PARK ) </a>, <a spellcheck="false"bbg-destination="mailto:pulp-list@redhat.com" class="" href="mailto:pulp-list@redhat.com" data-destination="mailto:pulp-list@redhat.com"> pulp-list@redhat.com</a><br>Subject: Re: [Pulp-list] pulp 3.7.3 sync with checksum error</div><br></div><div style="background: none 0% 0% repeat scroll white; color: black; font-family: Arial, "BB.Proportional"; font-size: small; white-space: normal;"><div><blockquote><div dir="ltr">Regarding the error message, I've observed the problem myself. and have filed an issue:<div><br></div><div><a spellcheck="false" bbg-destination="rte:bind" class=""href="https://pulp.plan.io/issues/8357" data-destination="rte:bind">https://pulp.plan.io/issues/8357</a><br><div><br></div><div>In the meantime, if you can patch the code, this should tell you give you more info:</div><div><br></div><div><a spellcheck="false" bbg-destination="rte:bind" class=""href="https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31"data-destination="rte:bind">https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31</a><br></div><div><div><br></div><div>David</div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 12:35 PM Brian Bouterse <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bmbouter@redhat.com" data-destination="mailto:rte:bind">bmbouter@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Did this happen inside a task? Did you see a traceback for it also?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 12:00 PM Bin Li (BLOOMBERG/ 120 PARK) <<a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:bli111@bloomberg.net" data-destination="mailto:rte:bind">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-size: small; font-family: "Courier New", Courier, "BB.FixedWidth"; white-space: pre-wrap;">The sync process gave an error "A file failed validation due to checksum". Is this error caused by remote repo? Is there a way to find out which file cause the issue?</div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> _______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div>_______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote></div></blockquote></div></blockquote><br></div></div></div></div></div></div></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></blockquote></div> _______________________________________________<br>Pulp-list mailing list<br><a spellcheck="false"bbg-destination="mailto:rte:bind" class="" href="mailto:Pulp-list@redhat.com" data-destination="mailto:rte:bind">Pulp-list@redhat.com</a><br><a spellcheck="false"bbg-destination="rte:bind" class="" href="https://listman.redhat.com/mailman/listinfo/pulp-list"data-destination="rte:bind">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote><br></div></div></div></div></div></blockquote></div></blockquote><br></div></div></div></div></div></div></body></html>