<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1>I found the cause. I had to modify /etc/httpd/conf.d/pulp_content.conf to support the symlinked location of the content and published files, which was causing errors like this one: The given path was above the root path: xsendfile: unable to find file: /data/var/lib/pulp/published/yum/master/yum_distributor/centos-6-x86_64-vmware-tools/1477638009.5/repodata/repomd.xml<o:p></o:p><o:p> </o:p><Location /pulp/content/><o:p></o:p> XSendFile on<o:p></o:p> XSendFilePath /data/var/lib/pulp/content<o:p></o:p> XSendFilePath /data/var/lib/pulp/published<o:p></o:p></Location><o:p></o:p><o:p> </o:p><o:p> </o:p><div>Dustin McNabb<o:p></o:p>| Verizon Enterprise Solutions<o:p></o:p>Cell: 775-453-4155<o:p></o:p>Pacific Time Zone<o:p></o:p></div><o:p> </o:p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>From: pulp-list-bounces@redhat.com [mailto:pulp-list-bounces@redhat.com] On Behalf Of Mcnabb, Dustin Sent: Saturday, October 29, 2016 10:36 AM To: pulp-list@redhat.com Subject: [E] [Pulp-list] Performed an upgrade of pulp 2.7.1-1 to pulp 2.10.0-1. Missing repodata now.<o:p></o:p></div></div><o:p> </o:p>After performing an upgrade from pulp 2.7 to pulp 2.10 in a lab environment I encountered a 404 error when trying to access the repo data doing a yum update. I navigated to this url in a browser and while I can view all of the packages I get the same 404 as for the repodata files when clicking on them in a web browser. The folder is there however. I tried doing a full resync and republish using the –force-full option for both, but this did not help. I also tried restarting pulp services including httpd. I also read the upgrade section of the release notes for every version of pulp released between 2.7 and 2.10 inclusive. I removed the python-semantic-version package as instructed in 2.8.5. I also tried to perform the data validation as instructed in 2.8.0 and documented here: <a href="https://raw.githubusercontent.com/pulp/pulp/pulp-2.8.0-1/playpen/mongoengine/README">https://raw.githubusercontent.com/pulp/pulp/pulp-2.8.0-1/playpen/mongoengine/README</a>. However, this kept failing due to a docker coredump, so I took a snapshot of the VM before proceeding with the upgrade. The db migration step took quite a long time (2 -3 hours) but then we have 200GB of repos on this pulp server. There were no glaring issues beyond a bunch of python deprecation warnings. Pulp-admin and https interfaces seem to be working normally, and I even found the actual repodata files under /var/lib/pulp/published and they seem normal as shown below. I am running centos 6.8 which was fully updated prior to the update of pulp. I hope I’ve covered all of the obvious things one would expect, but if I’ve missed anything or you need more information please let me know. Ultimately, I am just looking for advice on what might be the cause of this issue and how I should go about troubleshooting it further. Thanks for your time.<o:p></o:p><o:p> </o:p>[root@host homedir]# yum update<o:p></o:p><a href="https://%3cipaddr%3e/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml">https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml</a>: [Errno 14] HTTPS Error 404 - Not Found<o:p></o:p>Trying other mirror.<o:p></o:p>To address this issue please refer to the below knowledge base article<o:p></o:p><o:p> </o:p><a href="https://access.redhat.com/articles/1320623">https://access.redhat.com/articles/1320623</a> # I tried this. Removed yum cache and run yum clean all.<o:p></o:p><o:p> </o:p># Instructions for disabling repo redacted for brevity<o:p></o:p><o:p> </o:p>failure: repodata/repomd.xml from os: [Errno 256] No more mirrors to try.<o:p></o:p><a href="https://%3cipaddr%3e/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml">https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml</a>: [Errno 14] HTTPS Error 404 - Not Found<o:p></o:p><o:p> </o:p><o:p> </o:p>### repodata files do exist on the pulp server in the published dir despite 404s<o:p></o:p>[root@host:repodata]# pwd<o:p></o:p>/var/lib/pulp/published/yum/master/yum_distributor/redhat-7-x86_64-os/1477757560.86/repodata<o:p></o:p>[root@host:repodata]# ls -al<o:p></o:p>total 9536<o:p></o:p>drwxr-xr-x 2 apache apache 4096 Oct 29 16:17 ./<o:p></o:p>drwxr-xr-x 3 apache apache 438272 Oct 29 16:17 ../<o:p></o:p>-rw-r--r-- 1 apache apache 1909890 Oct 29 16:17 15a9c6779b3686f016a0b3b44c26883e7a346f9d-other.xml.gz<o:p></o:p>-rw-r--r-- 1 apache apache 1043185 Oct 29 16:17 32bfbe7576cffe021980a3657fab48a9b9a663e7-updateinfo.xml.gz<o:p></o:p>-rw-r--r-- 1 apache apache 3387052 Oct 29 16:17 580df8d0e09ea6c04363353c373557005ca0ad24-filelists.xml.gz<o:p></o:p>-rw-r--r-- 1 apache apache 2317010 Oct 29 16:17 7bd3bf3415e583c381a4880fe039165b983a1184-primary.xml.gz<o:p></o:p>-rw-r--r-- 1 apache apache 645418 Oct 29 16:17 aee7251856c47289f91a2f99c5dae2c8213a1e3a-comps.xml<o:p></o:p>lrwxrwxrwx 1 apache apache 126 Oct 29 16:17 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/52/6bf258c728e4bd7853dc59d0d8f23ecb6f23d1e49f76ca78169975e01e5a26/productid<o:p></o:p>-rw-r--r-- 1 apache apache 2019 Oct 29 16:17 repomd.xml<o:p></o:p><o:p> </o:p>Dustin McNabb<o:p></o:p></div></body></html>