[Pulp-list] ERROR: requested data type primary not available
Rolf Loudon
kalischt at gmail.com
Mon Aug 19 08:20:53 UTC 2019
Thank you for the suggestion. Sorry for the long delay in replying. We did try and re-publish the repo with the force full command but it made no difference.
We have found a workaround for our repository management and we now do not need to use pulp for it in this case. So I don’t have a need to pursue this request here, but thank you for the help.
As far as we got before trying another idea is that
in /var/lib/pulp/published/yum/master/yum_distributor/ <channel name>/
We saw this for a failing channel:
rhel-6-server-rpms]# ls -l */repodata/*
-rw-r--r--. 1 apache apache 1188540 Jul 30 06:11 1564465100.45/repodata/245ba6fe28e4a180b78cdd87f479af63cee47f4e-comps.xml
-rw-r--r--. 1 apache apache 45078695 Jul 30 06:03 1564465100.45/repodata/7fac8b0fefc059c50f4cc311f158eef6a9b34fd5-filelists.xml.gz
-rw-r--r--. 1 apache apache 3878183 Jul 30 06:10 1564465100.45/repodata/c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 57733120 Jul 30 06:03 1564465100.45/repodata/other.xml.gz
-rw-r--r--. 1 apache apache 40497152 Jul 30 06:03 1564465100.45/repodata/primary.xml.gz
-rw-r--r--. 1 apache apache 2159 Jul 30 03:25 1564465100.45/repodata/productid
-rw-r--r--. 1 apache apache 1129 Jul 30 06:29 1564465100.45/repodata/repomd.xml
And for a working one:
rhel-6-server-extras-rpms]# ls -l */repodata/*
-rw-r--r--. 1 apache apache 24475 Jul 29 13:05 1564405504.35/repodata/48c121111734b4da32d6404a178195b1aff90ace-primary.xml.gz
-rw-r--r--. 1 apache apache 124 Jul 29 13:05 1564405504.35/repodata/7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml
-rw-r--r--. 1 apache apache 45585 Jul 29 13:05 1564405504.35/repodata/97b983d84b21cca6f485389cb81856474502924a-other.xml.gz
-rw-r--r--. 1 apache apache 674921 Jul 29 13:05 1564405504.35/repodata/bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2-filelists.xml.gz
-rw-r--r--. 1 apache apache 37759 Jul 29 13:05 1564405504.35/repodata/db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 1813 Jul 29 13:05 1564405504.35/repodata/repomd.xml
But for the broken channel repomd.xml contains no reference to the primary or other xml files:
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo <http://linux.duke.edu/metadata/repo>" xmlns:rpm="http://linux.duke.edu/metadata/rpm <http://linux.duke.edu/metadata/rpm>"><revision>1564465100</revision>
<data type="updateinfo"><location href="repodata/c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be-updateinfo.xml.gz" /><timestamp>1564467055</timestamp><size>3878183</size><checksum type="sha1">c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be</checksum>
<open-size>19705102</open-size><open-checksum type="sha1">bbe90a0ff665d33c0a3e9302a397fd7f39ad6654</open-checksum></data>
<data type="group"><location href="repodata/245ba6fe28e4a180b78cdd87f479af63cee47f4e-comps.xml" /><timestamp>1564467062</timestamp><size>1188540</size><checksum type="sha1">245ba6fe28e4a180b78cdd87f479af63cee47f4e</checksum></data>
<data type="productid"><location href="repodata/productid" /><timestamp>1564457113</timestamp><size>2159</size><checksum type="sha1">625cf1b205833fcfb38a99c5ed94c3c25257379f</checksum></data>
</repomd>
While the working one has the repomd data with the correct references:
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo <http://linux.duke.edu/metadata/repo>" xmlns:rpm="http://linux.duke.edu/metadata/rpm <http://linux.duke.edu/metadata/rpm>"><revision>1564405504</revision>
<data type="filelists"><location href="repodata/bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2-filelists.xml.gz" /><timestamp>1564405505</timestamp><size>674921</size><checksum type="sha1">bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2</checksum><op
en-size>17892914</open-size><open-checksum type="sha1">15526bf2873fe3e6cac20664cfb7b011b2ad1541</open-checksum></data>
<data type="other"><location href="repodata/97b983d84b21cca6f485389cb81856474502924a-other.xml.gz" /><timestamp>1564405506</timestamp><size>45585</size><checksum type="sha1">97b983d84b21cca6f485389cb81856474502924a</checksum><open-size>2
42048</open-size><open-checksum type="sha1">6bc00f3aec0458704584b5b771271a22b4848de9</open-checksum></data>
<data type="primary"><location href="repodata/48c121111734b4da32d6404a178195b1aff90ace-primary.xml.gz" /><timestamp>1564405506</timestamp><size>24475</size><checksum type="sha1">48c121111734b4da32d6404a178195b1aff90ace</checksum><open-si
ze>244509</open-size><open-checksum type="sha1">23cb6df373205139fb3bf474f1ea3acb34007347</open-checksum></data>
<data type="updateinfo"><location href="repodata/db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6-updateinfo.xml.gz" /><timestamp>1564405508</timestamp><size>37759</size><checksum type="sha1">db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6</checksum><o
pen-size>217172</open-size><open-checksum type="sha1">49a1157c12b1764b387e3b83f0a2ee5413d63fc8</open-checksum></data>
<data type="group"><location href="repodata/7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml" /><timestamp>1564405508</timestamp><size>124</size><checksum type="sha1">7a747e6dc73c3ccef57c839995d90b6ef4c76092</checksum></data>
</repomd>
We are seeing the following error when some of the sync jobs are running:
pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) Finalizing failed
Jul 30 02:19:36 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) Traceback (most recent call last):
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 265, in process
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.finalize()
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 472, in finalize
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.file_lists_context.checksum)
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 132, in add_metadata_file_metadata
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) content = file_handle.read()
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 254, in read
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self._read(readsize)
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 313, in _read
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self._add_read_data( uncompress )
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 331, in _add_read_data
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.extrabuf = self.extrabuf[offset:] + data
Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) MemoryError
As I say we’ve solved our issue another way but am including the investigations as far as we got in the event it is useful elsewhere.
Regards
r.
> On 31 Jul 2019, at 2:03 am, Ina Panova <ipanova at redhat.com> wrote:
>
> Try to re-publish the repo with foce-full flag and see if the symlinks got fixed.
>
> Also check /var/lib/pulp/published, there you can find created repodata and symlinks pointing to the content.
>
>
> --------
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
> go instead where there is no path and leave a trail."
>
>
> On Sun, Jul 28, 2019 at 11:20 PM Rolf Loudon <kalischt at gmail.com <mailto:kalischt at gmail.com>> wrote:
>
> Hello
>
> Thanks for the assistance. The problem is more confusing than at first thought.
>
> If I go to http://xxxxxx/pulp/repos/rhel-6-server-rpms <http://xxxxxx/pulp/repos/rhel-6-server-rpms> in a browser I see the two links for ‘Packages’ and ‘repodata’.
>
> What I did not previously notice is that if I examine either link, or click on them, the URL used is not correct. It is http://xxxxxxx/pulp/repos/repodata <http://xxxxxxx/pulp/repos/repodata> which returns a 404.
> I tried manually entering http://xxxxxx/pulp/repos/rhel-6-server-rpms/repodata <http://xxxxxx/pulp/repos/rhel-6-server-rpms/repodata> and I get sensible information:
>
> 21dc6f5f63a58ea0c1f5afc1089b954a840234a6-comps.xml
> 5c8c97d6bbe41e935c933967a2a5f652ac138442-filelists.xml.gz
> 8c5ebae207e15589b4dd0362930f6986337be2d7-updateinfo.xml.gz
> other.xml.gz
> primary.xml.gz
> productid
> repomd.xml
>
> Correspondingly if I use http://xxxxxx/pulp/repos/rhel-6-server-rpms/Packages <http://xxxxxx/pulp/repos/rhel-6-server-rpms/Packages> I get a similar sensible output.
>
> I tested further and find that this behaviour is the same for all the categories, rhel-6-server-extras-rpms, rhel-6-server-optional-rpms, and rhel-6-server-supp-rpms and so on: the URL for ‘Packages’ and ‘repodata’ is not correct.
>
> What is further confusing is that using the spacewalk application to sync packages from my pulp server does in fact work for the other categories, just not for the Base packages:
>
> On the spacewalk host using the sync command:
>
> 1. For example the ‘Extras’ category:
>
> # spacewalk-repo-sync -c rhel6-server-extras-x86_64-pulp
> 20:48:54 ======================================
> 20:48:54 | Channel: rhel6-server-extras-x86_64-pulp
> 20:48:54 ======================================
> 20:48:54 Sync of channel started.
> 20:48:54 Repo URL: http://xxxxxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms <http://xxxxxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms>
> 20:48:55 Packages in repo: 97
> 20:48:59 No new packages to sync.
> 20:48:59 Repo http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms <http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms> has comps file 7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml.
> 20:48:59 Repo http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms <http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms> has 54 errata.
> 20:49:02 Sync of channel completed in 0:00:07.
> 20:49:02 Total time: 0:00:07
>
> There are no new packages in this category but as is shown there is no error returned
>
> 2. Using the base package:
>
> # spacewalk-repo-sync -c rhel6-x86_64-pulp
> 20:49:23 ======================================
> 20:49:23 | Channel: rhel6-x86_64-pulp
> 20:49:23 ======================================
> 20:49:23 Sync of channel started.
> 20:49:23 Repo URL: http://xxxxxxxxxxx/pulp/repos/rhel-6-server-rpms <http://xxxxxxxxxxx/pulp/repos/rhel-6-server-rpms>
> 20:49:24 ERROR: requested datatype primary not available
> 20:49:24 Sync of channel completed in 0:00:00.
> 20:49:24 Total time: 0:00:00
>
> The different behaviour between the Base set and the Extras, Supp, Optional, etc is further confusing and unexplained.
>
> So the essence of the problem now is that the URL provided for ‘Packages’ and ‘repodata’ in each category is wrong and does not include the category name.
>
> Any ideas?
>
> Many thanks
>
> r.
>
> On 26 Jul 2019, at 6:46 pm, Ina Panova <ipanova at redhat.com <mailto:ipanova at redhat.com>> wrote:
>
>> Hi Rolf,
>>
>> if you go to http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms <http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms> can you share the repodata with us?
>> The client seems to complain because of missing primary metadata file.
>>
>> --------
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>> go instead where there is no path and leave a trail."
>>
>>
>> On Wed, Jul 24, 2019 at 1:29 PM Rolf Loudon <kalischt at gmail.com <mailto:kalischt at gmail.com>> wrote:
>> Thanks
>>
>> Pulp install is via RPM and is version 2.14.0-1
>>
>> Yes, I am pointing the pulp create repo command to https://cdn.redhat.com/content/dist/rhel/server/6/6server/x86_64/os <https://cdn.redhat.com/content/dist/rhel/server/6/6server/x86_64/os>
>> I have seen the downloads being transferred and seemingly via Akamai (which I presume RedHat uses).
>>
>> R.
>>
>>> On 24 Jul 2019, at 9:17 pm, Dennis Kliban <dkliban at redhat.com <mailto:dkliban at redhat.com>> wrote:
>>>
>>> What version of Pulp are you using? Is the repository you are syncing publicly available?
>>>
>>> On Wed, Jul 24, 2019 at 2:52 AM Rolf Loudon <kalischt at gmail.com <mailto:kalischt at gmail.com>> wrote:
>>> Hello
>>>
>>> I have created and synced a repository in pulp.
>>>
>>> pulp-admin repo list shows it as follows:
>>>
>>> Id: rhel-6-server-rpms
>>>
>>> Display Name: None
>>>
>>> Description: None
>>>
>>> Content Unit Counts:
>>>
>>> Erratum: 4324
>>>
>>> Package Category: 10
>>>
>>> Package Group: 202
>>>
>>> Rpm: 20785
>>>
>>> Yum Repo Metadata File: 1
>>>
>>>
>>>
>>> I have a (old) spacewalk server on which I defined the above repo as a repository source. However when I try to sync the repo into spacewalk I get this:
>>>
>>> 06:44:11 Sync of channel started.
>>>
>>> 06:44:11 Repo URL: http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms <http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms>
>>> 06:44:11 ERROR: requested datatype primary not available
>>>
>>> 06:44:11 Sync of channel completed in 0:00:00.
>>>
>>> 06:44:11 Total time: 0:00:00
>>>
>>>
>>>
>>> To eliminate spacewalk as the source of the problem I configured a server that has never before used the pulp repo in /etc/yum.repos.d/pulp.repo :
>>>
>>>
>>>
>>> [rhel-base]
>>>
>>> name=Red Hat Enterprise Linux Pulp $releasever - $basearch
>>>
>>> baseurl=http://xxxxxxxxx/pulp/repos/rhel-6-server-rpms <http://xxxxxxxxx/pulp/repos/rhel-6-server-rpms>
>>> enabled=1
>>>
>>>
>>>
>>> When I run ‘yum repolist’ it fails in the same way:
>>>
>>>
>>>
>>> # yum repolist
>>>
>>> Loaded plugins: priorities, protectbase, refresh-packagekit, rhnplugin
>>>
>>> This system is receiving updates from RHN Classic or Red Hat Satellite.
>>>
>>> Error: requested datatype primary not available
>>>
>>>
>>>
>>> I have tried numerous times to recreate the repo in pulp and resync and so on.
>>>
>>> Any assistance much appreciated.
>>>
>>> r.
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/pulp-list <https://www.redhat.com/mailman/listinfo/pulp-list>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>> https://www.redhat.com/mailman/listinfo/pulp-list <https://www.redhat.com/mailman/listinfo/pulp-list>_______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list <https://www.redhat.com/mailman/listinfo/pulp-list>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190819/0ab8c738/attachment.htm>
More information about the Pulp-list
mailing list