[Pulp-list] Repo configured to use SHA checksums, but uploaded packages still get SHA256
Philipp Gassmann
gassmann at puzzle.ch
Wed Jun 18 09:32:11 UTC 2014
Hi Christina
Just replacing sha256 with sha does not work. (including deletion of
.py[o,c])
Then the generated hash in the pulp metadata is wrong.
And the hash in the repodata/primary.xml.gz is still sha256!
pulp-admin rpm repo content rpm --repo-id centos.5.x86_64.puzzle | grep
Data-Random -C 20
Checksum: d44cb4afbbdd2cca8cac107a19ff878847ec03e0
Checksumtype: sha
should be
# sha1sum perl-Data-Random-0.05-1.el5.noarch.rpm
4ad4b6eb87df53002f2c25223495a1503458420f
perl-Data-Random-0.05-1.el5.noarch.rpm
After # pulp-admin rpm repo publish run --repo-id centos.5.x86_64.puzzle
the repodata/...-primary.xml.gzl file still uses the sha256sum:
<name>perl-Data-Random</name>
<arch>noarch</arch>
<version epoch="0" ver="0.05" rel="1.el5"/>
<checksum type="sha256"
pkgid="YES">c45114bb7c1b2331e1bc352a691d3eaac1a14f3a6a5011eeae73c1eb8c8f86a2</checksum>
# sha256sum perl-Data-Random-0.05-1.el5.noarch.rpm
c45114bb7c1b2331e1bc352a691d3eaac1a14f3a6a5011eeae73c1eb8c8f86a2
perl-Data-Random-0.05-1.el5.noarch.rpm
Does it work differently on your system?
# rpm -qi pulp-admin-client
Version : 2.3.1
Regards,
Philipp Gassmann
Am 17.06.2014 17:43, schrieb Christina Plummer:
> Hi Philipp,
>
> I believe there were 3 lines that needed sha256 to be replaced by sha in
> package.py per my diff:
>
> 70: checksum = _calculate_checksum('sha', bundle.filename)
> 78: 'checksumtype' : 'sha',
> 143: unit_key['checksumtype'] = 'sha' # hardcoded to this in v1 so
> leaving this way for now
>
> Also, did you make sure to set aside the old .pyc and .pyo files and
> restart httpd before performing the upload?
> This will, of course, force all your uploaded packages to use 'sha' - so
> you would also need to change the checksum-type on your EL6 repos from
> sha256 to sha and republish.
> Christina
> On Tue, Jun 17, 2014 at 11:33 AM, Philipp Gassmann <gassmann at puzzle.ch
> <mailto:gassmann at puzzle.ch>> wrote:
>
> Hi,
>
> I tried no to modify the
> /usr/lib/python2.6/site-packages/pulp_rpm/extension/admin/upload/package.py
> as suggested by Christina Plummer.
> (on the separate build-server, not on the pulp server)
> just replacing sha256 with sha didn't work.
>
> _checksum_calculate_checksum(type,file) needs 'sha1' as type to generate
> a correct sha1 hash, but the checksum type for the pulp-metadata needs
> to be 'sha'.
>
> Now I managed to get the pulp-metadata right when i check on the command
> line:
> # pulp-admin rpm repo content rpm --repo-id centos.5.x86_64.puzzle |
> grep Data-Random -C 20
>
> Arch: noarch
> Buildhost: buildserver.not.puzzle.ch
> <http://buildserver.not.puzzle.ch/>
> Checksum: 4ad4b6eb87df53002f2c25223495a1503458420f
> Checksumtype: sha
> Description: A module used to generate random data. Useful mostly
> for test
> programs.
> Epoch: 0
> Filename: perl-Data-Random-0.05-1.el5.noarch.rpm
> License: Artistic/GPL
> Name: perl-Data-Random
> Provides: perl(Data::Random) = 0.05-0,
> perl(Data::Random::WordList) =
> 0.05-0, perl-Data-Random = 0.05-1.el5-0
> Release: 1.el5
> Requires: perl, perl(Carp), perl(Date::Calc), perl(Exporter),
> perl(GD)
> Vendor: Dag Apt Repository, http://dag.wieers.com/apt/
> Version: 0.05
>
> BUT then I run the repo publish command to generate the xml metadata
> # pulp-admin rpm repo publish run --repo-id centos.5.x86_64.puzzle
> +----------------------------------------------------------------------+
> Publishing Repository [centos.5.x86_64.puzzle]
> +----------------------------------------------------------------------+
>
> This command may be exited by pressing ctrl+c without affecting the
> actual
> operation on the server.
>
> Publishing distributions...
> [==================================================] 100%
> Distributions: 0/0 items
> ... completed
>
> Publishing packages...
> [==================================================] 100%
> Packages: 157/157 items
> ... completed
>
> Generating metadata
> [\]
> ... completed
>
> Publishing repository over HTTPS
> [-]
> ... completed
>
> Publishing repository over HTTP
> [\]
> ... completed
>
> AND it still generates the WRONG Hash to the repodata/primary.xml.gz:
>
> <package type="rpm">
> <name>perl-Data-Random</name>
> <arch>noarch</arch><version epoch="0" ver="0.05" rel="1.el5"/>
> <checksum type="sha256"
> pkgid="YES">c45114bb7c1b2331e1bc352a691d3eaac1a14f3a6a5011eeae73c1eb8c8f86a2</checksum>
> ...
>
> It looks completely broken, inconsistent on various levels
> (architecture/implementation/process).
> - checksum calculated per upload
> - checksum type per file in a repository
> - checksum type as repository option (OK) but without effect.
> - inconsistent checksum (type) between metadata in database and in xml
> repodata
> - type sha means hashlib.new('sha1')
> - Manual Package upload should be a core functionality of pulp and it is
> so broken
> - Bug is known since September last year
> - Bug is categorized to "user-experience"
> - Pulp is supposed to be a core part of Satellite 6
>
> Sorry for the rant, but I expect better quality from Red Hat.
> We want to use pulp as yum repository mirror and Server for RHEL/CentOS
> 5 & 6, stumbling over such a blocker is very disappointing.
>
> Regards,
> Philipp Gassmann
>
> Am 17.06.2014 15:36, schrieb Philipp Gassmann:
> > Hi Barnaby,
> >
> > This option (--checksum-type) is not present in the current stable
> > version of pulp (2.3.1) as Christina Plummer already mentioned in
> March.
> >
> > And as mentioned below, it is still hardcoded in the current
> master branch:
> > # Checksum
> > unit_key['checksumtype'] = 'sha256' # hardcoded to this in v1 so
> > leaving this way for now
> >
> >
> https://github.com/pulp/pulp_rpm/blame/master/extensions_admin/pulp_rpm/extensions/admin/upload/package.py#L178
> >
> >
> > from the details of repo list --detail (see below) it looks like
> > "Checksum Type" is a property of the distributor of the
> repository. That
> > makes sense to me, as the whole repository is to be consumed by a EL 5
> > client.
> > BTW. "Auto Publish: True" does not seem to work when manually
> uploading
> > a package (it did in pulp v1).
> >
> > Regards,
> > Philipp
> >
> > Am 17.06.2014 15:02, schrieb Barnaby Court:
> >> Hi, Pulp does support EL5 platforms. The trick is that you have
> to specify the checksum during the upload of RPM content units.
> Pulp treats files with a different checksum as completely different
> entities. Use the "pulp-admin rpm repo uploads rpm --checksum-type
> sha ..." for uploading the package for RHEL5. This allows you to
> override the default checksum type of sha256 whenever you upload an
> RPM. Upload the same rpm twice, once with --checksum-type sha for
> your el5 repo and once without the option specified or specified as
> sha256 for your EL6 repo. Regards,
> >>
> >> Barnaby
> >>
> >> ----- Original Message -----
> >> From: "Philipp Gassmann" <gassmann at puzzle.ch
> <mailto:gassmann at puzzle.ch>>
> >> To: "Martin Collins" <martin.collins at framestore.com
> <mailto:martin.collins at framestore.com>>, "Christina Plummer"
> <cplummer at gmail.com <mailto:cplummer at gmail.com>>, "Barnaby Court"
> <bcourt at redhat.com <mailto:bcourt at redhat.com>>
> >> Cc: pulp-list at redhat.com <mailto:pulp-list at redhat.com>
> >> Sent: Tuesday, June 17, 2014 4:51:57 AM
> >> Subject: Re: [Pulp-list] Repo configured to use SHA checksums,
> but uploaded packages still get SHA256
> >>
> >> Thanks for the replys!
> >> Unfortuately those workarounds are not really usable if you need to
> >> upload packages to existing repos for EL5 and EL6.
> >> I migrated the EL5 repos from pulp_v1 by initially syncing from those
> >> repos. Now I need to add additional packages to those repositories.
> >>
> >> I'm quite disappointed to find out that Pulp 2 does support
> repositories
> >> for EL5, even though RHEL 5 is listed as supported Platform.
> >>
> >> http://www.pulpproject.org/
> >> Supported Platforms:
> >> Fedora 18, Fedora 19, Red Hat Enterprise Linux 5, Red Hat
> Enterprise Linux 6
> >>
> >> The sha256 is still hardcoded:
> >>
> https://github.com/pulp/pulp_rpm/blob/master/extensions_admin/pulp_rpm/extensions/admin/upload/package.py
> >>
> >> The checksumtype is a repository option so the checksum should not be
> >> different for different packages of the same repository.
> >> Pulp 1 had the command to generate the metadata.
> >> Pulp 2 has the publish command which has a stage "generating
> metadata",
> >> there the checksum should be created.
> >> Am I wrong on that?
> >> Why does pulp 2.4 add an option "--checksum-type" to the upload
> command?
> >> Shouln't it use the repository checksum-type option?
> >> What is the checksum-type option of the repository for?
> >>
> >> pulp-admin rpm repo update --repo-id centos.5.x86_64.custom
> >> --checksum-type=sha1
> >>
> >> Id: centos.5.i386.updates
> >> Display Name: centos.5.i386.updates
> >> Description: Last checked by Puppet on Tue Jun 17
> 06:43:30 CEST 2014
> >> Content Unit Counts:
> >> Rpm: 408
> >> Notes:
> >> Importers:
> >> Config:
> >> Feed: http://mirror.switch.ch/ftp/mirror/centos/5/updates/i386/
> >> Id: yum_importer
> >> Importer Type Id: yum_importer
> >> Last Sync: 2014-05-21T11:54:49+02:00
> >> Repo Id: centos.5.i386.updates
> >> Scheduled Syncs:
> >> Distributors:
> >> Auto Publish: True
> >> Config:
> >> Checksum Type: sha1
> >> Http: True
> >> Https: True
> >> Relative URL: centos.5.i386.updates
> >> Distributor Type Id: yum_distributor
> >> Id: yum_distributor
> >> Last Publish: 2014-05-21T11:55:24+02:00
> >> Repo Id: centos.5.i386.updates
> >> Scheduled Publishes:
> >>
> >>
> >> The Bug Report is here:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1004580
> >>
> >>
> >> Am 16.06.2014 23:37, schrieb Martin Collins:
> >>> My workaround (as I have a large number of rhel6 and only a few for
> >>> rhel5) was to just sync from a file:/// feed URL for rhel5 then
> just put
> >>> the rpms in that dir, createrepo -s sha, then sync and continue
> normally
> >>> from there
> >>>
> >>> Cheers
> >>> Martin
> >>>
> >>> On 16 Jun 2014 22:33, "Christina Plummer" <cplummer at gmail.com
> <mailto:cplummer at gmail.com>
> >>> <mailto:cplummer at gmail.com <mailto:cplummer at gmail.com>>> wrote:
> >>>
> >>> Hi Philipp,
> >>>
> >>> My workaround was to modify the upload/package.py and explicitly
> >>> force it to use "sha" instead of "sha256". On my system,
> the file
> >>> was
> >>>
> /usr/lib/python2.6/site-packages/pulp_rpm/extension/admin/upload/package.py
> >>> . See attached diff.
> >>>
> >>> Be sure to set aside the package.pyc and package.pyo files
> as well,
> >>> and restart httpd. Then delete and re-upload the package to
> your repos.
> >>>
> >>> Belated public thanks to Ryan Bowlby for the pointer on this
> one.
> >>>
> >>> Christina
> >>>
> >>>
> >>> On Mon, Jun 16, 2014 at 10:19 AM, Philipp Gassmann
> >>> <gassmann at puzzle.ch <mailto:gassmann at puzzle.ch>
> <mailto:gassmann at puzzle.ch <mailto:gassmann at puzzle.ch>>> wrote:
> >>>
> >>> Hello Christina
> >>>
> >>> I just stumbled over the same issue. I uploaded the same
> noarch
> >>> package
> >>> to multiple repositories. now the installation of the
> package on
> >>> CentOS
> >>> 5 fails because of the wrong checksum.
> >>>
> >>> Did you find a solution or workaround?
> >>> I can install the package if I download it manually and
> install
> >>> it with
> >>> yum --nogpgcheck.
> >>>
> >>> Regards,
> >>> Philipp
> >>>
> >>>
> >>> Am 11.03.2014 14:26, schrieb Christina Plummer:
> >>> > Hi Barnaby,
> >>> >
> >>> > Thanks for your note. Unfortunately, --checksum-type
> doesn't
> >>> seem to be
> >>> > an option for the "pulp-admin rpm repo uploads rpm"
> command:
> >>> >
> >>> > CWC\cplummer1 at cbigdc-rhel6test ~ $ pulp-admin rpm repo
> uploads rpm
> >>> > --repo-id=q1-ol5-x86_64 --file
> >>> oracle-validated-1.0.0-34.el5.x86_64.rpm
> >>> > --checksum-type sha
> >>> > Command: rpm
> >>> > Description: uploads one or more RPMs into a repository
> >>> >
> >>> > Available Arguments:
> >>> >
> >>> > --repo-id - (required) unique identifier; only
> >>> alphanumeric, .,
> >>> > -, and _
> >>> > allowed
> >>> > --file, -f - full path to a file to upload; may be
> >>> specified multiple
> >>> > times for multiple files
> >>> > --dir, -d - full path to a directory
> containing files
> >>> to upload;
> >>> > may be
> >>> > specified multiple times for multiple
> >>> directories
> >>> > -v - display extra information about
> the upload
> >>> process
> >>> > --skip-existing - if specified, RPMs that already
> exist on
> >>> the server
> >>> > will not
> >>> > be uploaded
> >>> > The following options were specified but do not exist
> on the
> >>> command:
> >>> > --checksum-type
> >>> >
> >>> > I am using the RPMs for 2.3.1-1. Shouldn't it default
> to the
> >>> > checksum-type for the repository, if specified?
> >>> >
> >>> > Thanks,
> >>> > Christina
> >>> >
> >>> >
> >>> > On Mon, Mar 10, 2014 at 2:31 PM, Barnaby Court
> >>> <bcourt at redhat.com <mailto:bcourt at redhat.com>
> <mailto:bcourt at redhat.com <mailto:bcourt at redhat.com>>
> >>> > <mailto:bcourt at redhat.com <mailto:bcourt at redhat.com>
> <mailto:bcourt at redhat.com <mailto:bcourt at redhat.com>>>> wrote:
> >>> >
> >>> > Christina,
> >>> >
> >>> > The checksum on the RPM is set at upload time. Was
> >>> "--checksum-type
> >>> > SHA1" set on the "pulp-admin rpm repo uploads rpm"
> >>> command? If no
> >>> > checksum type is specified it defaults to SHA256.
> >>> >
> >>> > Regards,
> >>> > Barnaby
> >>> >
> >>> > ----- Original Message -----
> >>> > From: "Christina Plummer" <cplummer at gmail.com
> <mailto:cplummer at gmail.com>
> >>> <mailto:cplummer at gmail.com <mailto:cplummer at gmail.com>>
> >>> > <mailto:cplummer at gmail.com
> <mailto:cplummer at gmail.com> <mailto:cplummer at gmail.com
> <mailto:cplummer at gmail.com>>>>
> >>> > To: pulp-list at redhat.com
> <mailto:pulp-list at redhat.com> <mailto:pulp-list at redhat.com
> <mailto:pulp-list at redhat.com>>
> >>> <mailto:pulp-list at redhat.com
> <mailto:pulp-list at redhat.com> <mailto:pulp-list at redhat.com
> <mailto:pulp-list at redhat.com>>>
> >>> > Sent: Monday, March 10, 2014 1:42:14 PM
> >>> > Subject: [Pulp-list] Repo configured to use SHA
> checksums,
> >>> but
> >>> > uploaded packages still get SHA256
> >>> >
> >>> > On Pulp 2.1.3, I had run into issues with sha256
> checksums
> >>> being
> >>> > created for some of the repo metadata files, even
> though
> >>> the repo
> >>> > was configured to use sha instead.
> >>> >
> >>> > On Pulp 2.3.1, this seems to have been fixed...
> however, I
> >>> am now
> >>> > seeing that packages manually uploaded into a repo are
> >>> still getting
> >>> > a sha256 checksum in the repodata, which means
> that RHEL5/OL5
> >>> > clients can't handle them. Packages that were
> synced, or
> >>> copied from
> >>> > another repository, have the correct checksum type.
> >>> >
> >>> > OS: RHEL 6.5 x86_64
> >>> > Pulp: 2.3.1-1 (RPMs from Pulp repo)
> >>> > Steps to reproduce:
> >>> > 1. Created "live" repo fed from Oracle public yum
> >>> repositories,
> >>> > checksum-type=sha, retain-old-count=1
> >>> > 2. Created "q1" repo, checksum-type=sha
> >>> > 3. Copied rpms from "live" to "q1
> >>> > 4. Deleted "known-broken" versions of
> >>> oracle-validated-1.1.0 package
> >>> > 5. Uploaded "known-working" version of
> >>> oracle-validated-1.0.0 package
> >>> > 6. Published repo. Every package except for
> >>> oracle-validated-1.0.0
> >>> > has a 'sha' checksum; oracle-validated-1.0.0 alone
> has a
> >>> 'sha256'
> >>> > checksum.
> >>> >
> >>> > Is this a bug?
> >>> >
> >>> > Thanks,
> >>> > Christina
> >>> >
> >>> > _______________________________________________
> >>> > Pulp-list mailing list
> >>> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> <mailto:Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>>
> >>> <mailto:Pulp-list at redhat.com
> <mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com
> <mailto:Pulp-list at redhat.com>>>
> >>> > https://www.redhat.com/mailman/listinfo/pulp-list
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Pulp-list mailing list
> >>> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> <mailto:Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>>
> >>> > https://www.redhat.com/mailman/listinfo/pulp-list
> >>> >
> >>>
> >>>
> >>> --
> >>> Philipp Gassmann
> >>>
> >>> Puzzle ITC GmbH
> >>> http://www.puzzle.ch <http://www.puzzle.ch/>
> >>>
> >>> Telefon +41 31 370 22 00
> <tel:%2B41%2031%20370%2022%2000> <tel:%2B41%2031%20370%2022%2000>
> >>> Werfen Sie einen Blick in unseren Blog:
> >>> http://www.puzzle.ch/blog
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Pulp-list mailing list
> >>> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> <mailto:Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>>
> >>> https://www.redhat.com/mailman/listinfo/pulp-list
> >>>
> >>
> >>
> >
> >
>
>
> --
> Philipp Gassmann
>
> Puzzle ITC GmbH
> http://www.puzzle.ch <http://www.puzzle.ch/>
>
> Telefon +41 31 370 22 00 <tel:%2B41%2031%20370%2022%2000>
> Direkt +41 31 370 22 02 <tel:%2B41%2031%20370%2022%2002>
> Fax +41 31 370 22 01 <tel:%2B41%2031%20370%2022%2001>
>
> Werfen Sie einen Blick in unseren Blog:
> http://www.puzzle.ch/blog
>
>
--
Philipp Gassmann
Puzzle ITC GmbH
http://www.puzzle.ch
Telefon +41 31 370 22 00
Direkt +41 31 370 22 02
Fax +41 31 370 22 01
Werfen Sie einen Blick in unseren Blog:
http://www.puzzle.ch/blog
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20140618/625355b3/attachment.sig>
More information about the Pulp-list
mailing list