[Pulp-list] [E] Performed an upgrade of pulp 2.7.1-1 to pulp 2.10.0-1. Missing repodata now.

Sat Oct 29 19:28:35 UTC 2016

I found the cause. I had to modify /etc/httpd/conf.d/pulp_content.conf to support the symlinked location of the content and published files, which was causing errors like this one: The given path was above the root path: xsendfile: unable to find file: /data/var/lib/pulp/published/yum/master/yum_distributor/centos-6-x86_64-vmware-tools/1477638009.5/repodata/repomd.xml

<Location /pulp/content/>

    XSendFile on

    XSendFilePath /data/var/lib/pulp/content

    XSendFilePath /data/var/lib/pulp/published

</Location>

Dustin McNabb
| Verizon Enterprise Solutions
Cell: 775-453-4155
Pacific Time Zone

From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com] On Behalf Of Mcnabb, Dustin
Sent: Saturday, October 29, 2016 10:36 AM
To: pulp-list at redhat.com
Subject: [E] [Pulp-list] Performed an upgrade of pulp 2.7.1-1 to pulp 2.10.0-1. Missing repodata now.

After performing an upgrade from pulp 2.7 to pulp 2.10 in a lab environment I encountered a 404 error when trying to access the repo data doing a yum update. I navigated to this url in a browser and while I can view all of the packages I get the same 404 as for the repodata files when clicking on them in a web browser. The folder is there however. I tried doing a full resync and republish using the -force-full option for both, but this did not help. I also tried restarting pulp services including httpd. I also read the upgrade section of the release notes for every version of pulp released between 2.7 and 2.10 inclusive. I removed the python-semantic-version package as instructed in 2.8.5. I also tried to perform the data validation as instructed in 2.8.0 and documented here: https://raw.githubusercontent.com/pulp/pulp/pulp-2.8.0-1/playpen/mongoengine/README. However, this kept failing due to a docker coredump, so I took a snapshot of the VM before proceeding with the upgrade. The db migration step took quite a long time (2 -3 hours) but then we have 200GB of repos on this pulp server. There were no glaring issues beyond a bunch of python deprecation warnings. Pulp-admin and https interfaces seem to be working normally, and I even found the actual repodata files under /var/lib/pulp/published and they seem normal as shown below.  I am running centos 6.8 which was fully updated prior to the update of pulp. I hope I've covered all of the obvious things one would expect, but if I've missed anything or you need more information please let me know. Ultimately, I am just looking for advice on what might be the cause of this issue and how I should go about troubleshooting it further. Thanks for your time.

[root at host homedir]# yum update
https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml<https://%3cipaddr%3e/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml>: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623                       # I tried this.  Removed yum cache and run yum clean all.

# Instructions for disabling repo redacted for brevity

failure: repodata/repomd.xml from os: [Errno 256] No more mirrors to try.
https://<ipaddr>/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml<https://%3cipaddr%3e/pulp/repos/redhat/7/x86_64/os/repodata/repomd.xml>: [Errno 14] HTTPS Error 404 - Not Found

### repodata files do exist on the pulp server in the published dir despite 404s
[root at host:repodata]# pwd
/var/lib/pulp/published/yum/master/yum_distributor/redhat-7-x86_64-os/1477757560.86/repodata
[root at host:repodata]# ls -al
total 9536
drwxr-xr-x 2 apache apache    4096 Oct 29 16:17 ./
drwxr-xr-x 3 apache apache  438272 Oct 29 16:17 ../
-rw-r--r-- 1 apache apache 1909890 Oct 29 16:17 15a9c6779b3686f016a0b3b44c26883e7a346f9d-other.xml.gz
-rw-r--r-- 1 apache apache 1043185 Oct 29 16:17 32bfbe7576cffe021980a3657fab48a9b9a663e7-updateinfo.xml.gz
-rw-r--r-- 1 apache apache 3387052 Oct 29 16:17 580df8d0e09ea6c04363353c373557005ca0ad24-filelists.xml.gz
-rw-r--r-- 1 apache apache 2317010 Oct 29 16:17 7bd3bf3415e583c381a4880fe039165b983a1184-primary.xml.gz
-rw-r--r-- 1 apache apache  645418 Oct 29 16:17 aee7251856c47289f91a2f99c5dae2c8213a1e3a-comps.xml
lrwxrwxrwx 1 apache apache     126 Oct 29 16:17 productid -> /var/lib/pulp/content/units/yum_repo_metadata_file/52/6bf258c728e4bd7853dc59d0d8f23ecb6f23d1e49f76ca78169975e01e5a26/productid
-rw-r--r-- 1 apache apache    2019 Oct 29 16:17 repomd.xml

Dustin McNabb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20161029/6a469f32/attachment.htm>