[Spacewalk-list] Support for LZMA-compressed comps.xml

Fri Jan 30 16:21:44 UTC 2015

Since I don't have Spacewalk 2.2 or one of the nightly versions
installed here, I can't say for sure that the issue does not exist in
them. Note though that I was able to sync the repository successfully
even before I made the change. The problem is that spacewalk is still
storing the compressed comps.xml but serves it to yum clients as if it
was not compressed.

Can you confirm that the comps.xml as stored in the satellite/rhn/comps
directory for the channel is the uncompressed XML file? I found that it
would always be the LZMA-compressed version and be named
503597f15d6c693a96558f46fbd8ebb5853b92ef528de2a7d55a4dd021162873-comps-f21.xml.xz.
However, the repomd.xml file indicates that the comps file is the
uncompressed form. As best I can tell, spacewalk is serving the
compressed comps.xml as if it were uncompressed (as far as yum is
concerned). This causes yum to expect it to be an uncompressed XML
document and it thus generates an error indicating it is malformed. You
would only notice this if you actually subscribed a system to the
channel and tried to do some sort of yum operation. From the perspective
of Spacewalk itself, everything seems to work fine.

The fix I found was to make sure spacewalk stores the uncompressed one,
since that is what it seems to indicate is present in repomd.xml. This
appears to be consistent with previous distros where the comps.xml is
gzip compressed but is stored/served by spacewalk in uncompressed form.

Brian

On 01/30/2015 05:20 AM, Tomas Lestach wrote:
> I was able to sync fc21 repo on Spacewalk nightly together with the comps file.
> Feel free to check [1] for more information.
>
> Even if  Brian installed pyliblzma, it's possible there were more changes done
> since Spacewalk 2.1 that affect the issue.
>
> Regards,
> --
> Tomas Lestach
> Red Hat Satellite Engineering, Red Hat
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1144008
>
>
> ----- Original Message -----
>> From: "Stephen Herr" <sherr at redhat.com>
>> To: spacewalk-list at redhat.com
>> Sent: Thursday, January 29, 2015 4:07:08 PM
>> Subject: Re: [Spacewalk-list] Support for LZMA-compressed comps.xml
>>
>> Here you go Tomas:
>>
>> https://dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/repodata/
>>
>> Take a look at *-comps-f21.xml.xz
>>
>> -Stephen
>>
>> On 01/29/2015 06:23 AM, Tomas Lestach wrote:
>>> Would you share a public repo with lzma compressed comps?
>>>
>>> Regards,
>>> --
>>> Tomas Lestach
>>> Red Hat Satellite Engineering, Red Hat
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Brian Buesker" <bbuesker at qti.qualcomm.com>
>>>> To: spacewalk-list at redhat.com
>>>> Sent: Wednesday, January 28, 2015 9:53:48 PM
>>>> Subject: [Spacewalk-list] Support for LZMA-compressed comps.xml
>>>>
>>>>
>>>> I've started trying to bring online a Fedora 21 distro/kickstart
>>>> profile against our Spacewalk 2.1 server. Unlike others who have
>>>> had
>>>> trouble getting the fedora installer generation to work, it mostly
>>>> seems to be working for me. I'm not sure if that is because I am
>>>> using 2.1 instead of 2.2 or if there is some other explanation.
>>>>
>>>> However, one thing I ran into is that when I created a Fedora 21
>>>> Everything channel, Spacewalk ended up not handling the comps.xml
>>>> properly. In the upstream yum repo, the comps.xml is actually LZMA
>>>> compressed. It does not look like Spacewalk decompresses this when
>>>> staging the comps in the local file system. Thus, when a client
>>>> fetches the repo metadata, it gets a file named comps.xml that is
>>>> actually compressed and thus it complains about the XML not being
>>>> well formed.
>>>>
>>>> I looked at the Spacewalk git repo and I see that an RPM
>>>> dependency
>>>> was added for pyliblzma. I already have that installed on my
>>>> system,
>>>> but despite that, Spacewalk still does not handle the compressed
>>>> comps properly. I dug further into this and found that two python
>>>> modules need to be updated, namely reposync.py and fileutils.py.
>>>>
>>>> I would provide a patch with the fixes, although I have to go
>>>> through
>>>> a somewhat lengthy procedure before I can get permission from my
>>>> employer to release the patch. Therefore, I'll describe the
>>>> necessary changes in case others want to make the same ones (or
>>>> even
>>>> better, get this updated in git).
>>>>
>>>>
>>>>      1. In reposync.py, the list of suffixes in import_groups()
>>>>      needs
>>>>      to include '.xz'.
>>>>      2. In fileutils.py, decompress_open() needs to be updated to
>>>>      use
>>>>      the lzma.LZMAFile() function on files ending in '.xz' (and of
>>>>      course an import is needed as well).
>>>>
>>>>
>>>> Let me know if you have any questions.
>>>>
>>>>
>>>> Thanks,
>>>> Brian Buesker
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>