[Spacewalk-list] Support for LZMA-compressed comps.xml
Stephen Herr
sherr at redhat.com
Fri Feb 13 21:10:44 UTC 2015
Thanks guys, hopefully this looks reasonable:
https://github.com/spacewalkproject/spacewalk/commit/76883065ff72449726adc82d081ae681f2a5bb59
RHEL 6 does not have pyliblzma installed by default but it's available
in EPEL so I had to add a requirement to the spec file. RHEL 5 doesn't
have it available but it should be fine since we shouldn't be releasing
Spacewalk 2.3 for RHEL 5 anyway.
-Stephen
On 02/03/2015 01:04 PM, Michael Guidero wrote:
> I have Spacewalk 2.2 and this does occur with it as well. I have the
> issue with the Scientific Linux repository.
>
> http://ftp.scientificlinux.org/linux/scientific/7x/x86_64/os/repodata/
>
> Spacewalk delivers a compressed comps.xml file to my client systems with
> no .xz extension.
>
> [root at swtest01 sl-7-x86_64-7x]# pwd
> /var/cache/yum/x86_64/7/sl-7-x86_64-7x
> [root at swtest01 sl-7-x86_64-7x]# file comps.xml
> comps.xml: XZ compressed data
>
> Yum complains:
>
> [root at swtest01 sl-7-x86_64-7x]# yum group list
> Loaded plugins: rhnplugin, versionlock
> This system is receiving updates from RHN Classic or Red Hat Satellite.
> There is no installed groups file.
> Maybe run: yum groups mark convert (see man yum)
> Failed to add groups file for repository: sl-7-x86_64-7x - comps file is
> empty/damaged
> Available environment groups:
> ...
>
>
>
>
>
> On Fri, Jan 30, 2015 at 8:21 AM, Brian Buesker
> <bbuesker at qti.qualcomm.com <mailto:bbuesker at qti.qualcomm.com>> wrote:
>
> Since I don't have Spacewalk 2.2 or one of the nightly versions
> installed here, I can't say for sure that the issue does not exist in
> them. Note though that I was able to sync the repository successfully
> even before I made the change. The problem is that spacewalk is still
> storing the compressed comps.xml but serves it to yum clients as if it
> was not compressed.
>
> Can you confirm that the comps.xml as stored in the satellite/rhn/comps
> directory for the channel is the uncompressed XML file? I found that it
> would always be the LZMA-compressed version and be named
> 503597f15d6c693a96558f46fbd8ebb5853b92ef528de2a7d55a4dd021162873-comps-f21.xml.xz.
> However, the repomd.xml file indicates that the comps file is the
> uncompressed form. As best I can tell, spacewalk is serving the
> compressed comps.xml as if it were uncompressed (as far as yum is
> concerned). This causes yum to expect it to be an uncompressed XML
> document and it thus generates an error indicating it is malformed. You
> would only notice this if you actually subscribed a system to the
> channel and tried to do some sort of yum operation. From the perspective
> of Spacewalk itself, everything seems to work fine.
>
> The fix I found was to make sure spacewalk stores the uncompressed one,
> since that is what it seems to indicate is present in repomd.xml. This
> appears to be consistent with previous distros where the comps.xml is
> gzip compressed but is stored/served by spacewalk in uncompressed form.
>
> Brian
>
> On 01/30/2015 05:20 AM, Tomas Lestach wrote:
> > I was able to sync fc21 repo on Spacewalk nightly together with
> the comps file.
> > Feel free to check [1] for more information.
> >
> > Even if Brian installed pyliblzma, it's possible there were more
> changes done
> > since Spacewalk 2.1 that affect the issue.
> >
> > Regards,
> > --
> > Tomas Lestach
> > Red Hat Satellite Engineering, Red Hat
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1144008
> >
> >
> > ----- Original Message -----
> >> From: "Stephen Herr" <sherr at redhat.com <mailto:sherr at redhat.com>>
> >> To: spacewalk-list at redhat.com <mailto:spacewalk-list at redhat.com>
> >> Sent: Thursday, January 29, 2015 4:07:08 PM
> >> Subject: Re: [Spacewalk-list] Support for LZMA-compressed comps.xml
> >>
> >> Here you go Tomas:
> >>
> >>
> https://dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/repodata/
> >>
> >> Take a look at *-comps-f21.xml.xz
> >>
> >> -Stephen
> >>
> >> On 01/29/2015 06:23 AM, Tomas Lestach wrote:
> >>> Would you share a public repo with lzma compressed comps?
> >>>
> >>> Regards,
> >>> --
> >>> Tomas Lestach
> >>> Red Hat Satellite Engineering, Red Hat
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Brian Buesker" <bbuesker at qti.qualcomm.com
> <mailto:bbuesker at qti.qualcomm.com>>
> >>>> To: spacewalk-list at redhat.com <mailto:spacewalk-list at redhat.com>
> >>>> Sent: Wednesday, January 28, 2015 9:53:48 PM
> >>>> Subject: [Spacewalk-list] Support for LZMA-compressed comps.xml
> >>>>
> >>>>
> >>>> I've started trying to bring online a Fedora 21 distro/kickstart
> >>>> profile against our Spacewalk 2.1 server. Unlike others who have
> >>>> had
> >>>> trouble getting the fedora installer generation to work, it mostly
> >>>> seems to be working for me. I'm not sure if that is because I am
> >>>> using 2.1 instead of 2.2 or if there is some other explanation.
> >>>>
> >>>> However, one thing I ran into is that when I created a Fedora 21
> >>>> Everything channel, Spacewalk ended up not handling the comps.xml
> >>>> properly. In the upstream yum repo, the comps.xml is actually LZMA
> >>>> compressed. It does not look like Spacewalk decompresses this when
> >>>> staging the comps in the local file system. Thus, when a client
> >>>> fetches the repo metadata, it gets a file named comps.xml that is
> >>>> actually compressed and thus it complains about the XML not being
> >>>> well formed.
> >>>>
> >>>> I looked at the Spacewalk git repo and I see that an RPM
> >>>> dependency
> >>>> was added for pyliblzma. I already have that installed on my
> >>>> system,
> >>>> but despite that, Spacewalk still does not handle the compressed
> >>>> comps properly. I dug further into this and found that two python
> >>>> modules need to be updated, namely reposync.py and fileutils.py.
> >>>>
> >>>> I would provide a patch with the fixes, although I have to go
> >>>> through
> >>>> a somewhat lengthy procedure before I can get permission from my
> >>>> employer to release the patch. Therefore, I'll describe the
> >>>> necessary changes in case others want to make the same ones (or
> >>>> even
> >>>> better, get this updated in git).
> >>>>
> >>>>
> >>>> 1. In reposync.py, the list of suffixes in import_groups()
> >>>> needs
> >>>> to include '.xz'.
> >>>> 2. In fileutils.py, decompress_open() needs to be updated to
> >>>> use
> >>>> the lzma.LZMAFile() function on files ending in '.xz' (and of
> >>>> course an import is needed as well).
> >>>>
> >>>>
> >>>> Let me know if you have any questions.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Brian Buesker
> >>>>
> >>>> _______________________________________________
> >>>> Spacewalk-list mailing list
> >>>> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> >>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
> >>> _______________________________________________
> >>> Spacewalk-list mailing list
> >>> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> >>> https://www.redhat.com/mailman/listinfo/spacewalk-list
> >>>
> >> _______________________________________________
> >> Spacewalk-list mailing list
> >> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> >> https://www.redhat.com/mailman/listinfo/spacewalk-list
> >>
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
>
> --
>
> Michael Guidero
> Sococo IT
> 650-265-7013 Ext 1000
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
More information about the Spacewalk-list
mailing list