<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>After performing an upgrade from pulp 2.7 to pulp 2.10 in a lab environment I encountered a 404 error when trying to access the repo data doing a yum update. I navigated to this url in a browser and while I can view all of the packages I get the same 404 as for the repodata files when clicking on them in a web browser. The folder is there however. I tried doing a full resync and republish using the –force-full option for both, but this did not help. I also tried restarting pulp services including httpd. I also read the upgrade section of the release notes for every version of pulp released between 2.7 and 2.10 inclusive. I removed the python-semantic-version package as instructed in 2.8.5. I also tried to perform the data validation as instructed in 2.8.0 and documented here: <a href="https://raw.githubusercontent.com/pulp/pulp/pulp-2.8.0-1/playpen/mongoengine/README">https://raw.githubusercontent.com/pulp/pulp/pulp-2.8.0-1/playpen/mongoengine/README</a>. However, this kept failing due to a docker coredump, so I took a snapshot of the VM before proceeding with the upgrade. The db migration step took quite a long time (2 -3 hours) but then we have 200GB of repos on this pulp server. There were no glaring issues beyond a bunch of python deprecation warnings. Pulp-admin and https interfaces seem to be working normally, and I even found the actual repodata files under /var/lib/pulp/published and they seem normal as shown below. I am running centos 6.8 which was fully updated prior to the update of pulp. I hope I’ve covered all of the obvious things one would expect, but if I’ve missed anything or you need more information please let me know. Ultimately, I am just looking for advice on what might be the cause of this issue and how I should go about troubleshooting it further. Thanks for your time.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[root@host homedir]# yum update<o:p></o:p></p><p class=MsoNormal>https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found<o:p></o:p></p><p class=MsoNormal>Trying other mirror.<o:p></o:p></p><p class=MsoNormal>To address this issue please refer to the below knowledge base article<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><a href="https://access.redhat.com/articles/1320623">https://access.redhat.com/articles/1320623</a> # I tried this. Removed yum cache and run yum clean all.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal># Instructions for disabling repo redacted for brevity<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>failure: repodata/repomd.xml from os: [Errno 256] No more mirrors to try.<o:p></o:p></p><p class=MsoNormal>https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>### repodata files do exist on the pulp server in the published dir despite 404s<o:p></o:p></p><p class=MsoNormal>[root@host:repodata]# pwd<o:p></o:p></p><p class=MsoNormal>/var/lib/pulp/published/yum/master/yum_distributor/redhat-7-x86_64-os/1477757560.86/repodata<o:p></o:p></p><p class=MsoNormal>[root@host:repodata]# ls -al<o:p></o:p></p><p class=MsoNormal>total 9536<o:p></o:p></p><p class=MsoNormal>drwxr-xr-x 2 apache apache 4096 Oct 29 16:17 ./<o:p></o:p></p><p class=MsoNormal>drwxr-xr-x 3 apache apache 438272 Oct 29 16:17 ../<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 1909890 Oct 29 16:17 15a9c6779b3686f016a0b3b44c26883e7a346f9d-other.xml.gz<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 1043185 Oct 29 16:17 32bfbe7576cffe021980a3657fab48a9b9a663e7-updateinfo.xml.gz<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 3387052 Oct 29 16:17 580df8d0e09ea6c04363353c373557005ca0ad24-filelists.xml.gz<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 2317010 Oct 29 16:17 7bd3bf3415e583c381a4880fe039165b983a1184-primary.xml.gz<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 645418 Oct 29 16:17 aee7251856c47289f91a2f99c5dae2c8213a1e3a-comps.xml<o:p></o:p></p><p class=MsoNormal>lrwxrwxrwx 1 apache apache 126 Oct 29 16:17 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/52/6bf258c728e4bd7853dc59d0d8f23ecb6f23d1e49f76ca78169975e01e5a26/productid<o:p></o:p></p><p class=MsoNormal>-rw-r--r-- 1 apache apache 2019 Oct 29 16:17 repomd.xml<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Dustin McNabb<o:p></o:p></p></div></body></html>