[Spacewalk-list] After Spacewalk serverrename/selfsigned SSL-issues on yum-updates for CentOS

Thu Jun 13 07:46:10 UTC 2013

Dear all,

once again I'm puzzling about some strange issue and am hoping for some
valued input I experienced in the past on this list:

The gist (for the hasty reader:):
CentOS yum/curl rejects my Spacewalk' servers certificate as "Bad
certificate received"
while a check with openssl s_client works.
Fedoras don't show the issue at all.

Any ideas?

Best
-Jonathan

#####
Here is a bit more length explanation, but I want to show what works to
exclude some point-of-errors

So I renamed my Spacewalk-Server some while ago,
and in/after the renaming (DNS etc reports clean name, the server has
the name, all fine),
I updated the Spacewalk servers' Certs, by using my own self-signed
root-CA (which now not only exists in /etc/pki/tls/certs/ca-bundle* but
of course also in /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT ).

Everything works fine (I of course had to deploy the rootCA to my
clients, update the server-name in up2date etc),
except for
yum update - commands on CentOS boxes.

And this is the issue.
Fedora (16,17,18) works fine, with identical steps of change-deployment.

Debugging the yum-commands shows (domain+email slightly modified) (yum
debug=10, on CentOS 5/6)
"""
[...]
2013-06-13 09:19:33,487 attempt 1/1:
https://ibis-spacewalk.xx-muenchen.de/XMLRPC/GET-REQ/centos6-x86_64/repodata/repomd.xml
INFO:urlgrabber:attempt 1/1:
https://ibis-spacewalk.xx-muenchen.de/XMLRPC/GET-REQ/centos6-x86_64/repodata/repomd.xml
2013-06-13 09:19:33,487 opening local file
"/var/cache/yum/x86_64/6/centos6-x86_64/repomdKZp7dWtmp.xml" with mode wb
INFO:urlgrabber:opening local file
"/var/cache/yum/x86_64/6/centos6-x86_64/repomdKZp7dWtmp.xml" with mode wb
* About to connect() to ibis-spacewalk.xx-muenchen.de port 443 (#0)
*   Trying xx.107.218.92... * connected
* Connected to ibis-spacewalk.xx-muenchen.de (xx.107.218.92) port 443 (#0)
* warning: CURLOPT_CAPATH not a directory
(/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT)
*   CAfile: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
   CApath: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
* Bad certificate received. Subject =
'E=me at xx-muenchen.de,CN=ibis-spacewalk.xx-muenchen.de,OU=IBIS,O=Helmholtz-Zentrum
Muenchen GmbH,L=Munich,ST=Bavaria,C=DE', Issuer =
'E=me at xx-muenchen.de,CN=IBIS-Root-CA,OU=IBIS,O=Helmholtz-Zentrum
Muenchen GmbH,L=Munich,ST=Bavaria,C=DE'
* NSS error -8182
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
2013-06-13 09:19:33,689 exception: [Errno 14] Peer cert cannot be
verified or peer cert invalid
INFO:urlgrabber:exception: [Errno 14] Peer cert cannot be verified or
peer cert invalid
2013-06-13 09:19:33,690 retries exceeded, re-raising
INFO:urlgrabber:retries exceeded, re-raising
Error: Cannot retrieve repository metadata (repomd.xml) for repository:
centos6-x86_64. Please verify its path and try again
"""

So I googled the messages and re-discovered it uses curl.
Trying curl 'naked' on my server-url:
"""
curl --cacert /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
https://ibis-spacewalk.xx-muenchen.de -v
* About to connect() to ibis-spacewalk.xx-muenchen.de port 443 (#0)
*   Trying xx.107.218.92... connected
* Connected to ibis-spacewalk.xx-muenchen.de (146.107.218.92) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
   CApath: none
* Bad certificate received. Subject =
'E=me at xx-muenchen.de,CN=ibis-spacewalk.xx-muenchen.de,OU=IBIS,O=Helmholtz-Zentrum
Muenchen GmbH,L=Munich,ST=Bavaria,C=DE', Issuer =
'E=me at xx-muenchen.de,CN=IBIS-Root-CA,OU=IBIS,O=Helmholtz-Zentrum
Muenchen GmbH,L=Munich,ST=Bavaria,C=DE'
* NSS error -8182
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
curl: (60) Peer certificate cannot be authenticated with known CA
certificates
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
  of Certificate Authority (CA) public keys (CA certs). If the default
  bundle file isn't adequate, you can specify an alternate file
  using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
  the bundle, the certificate verification probably failed due to a
  problem with the certificate (it might be expired, or the name might
  not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
  the -k (or --insecure) option.
"""

With the -k version it works, no questions asked.
"""
openssl s_client -connect ibis-spacewalk.xx-muenchen.de:443
[...finally returns...]
Verify return code: 0 (ok)
"""
and thus works, and is happy with the results/the ca-certs.

The crazy thing is - as I mentioned - that the identical requests (from
yum and curl)
work on Fedoras (16-18).
The only real difference are the libcurl/curl versions:
Fedora17 uses 7.24.0 (for both)
CentOS 6.4 uses 7.19.7

I am a bit out of ideas here
  - I also added the rootCA to the NSS store (since this is also loaded
in curl-Standalone)
but that also doesn't change a thing.

Any input would be appreciated,

with best regards
-Jonathan

--
Jonathan Hoser, M.Sc.
Institute of Bioinformatics and System Biology

WWW: http://mips.xx-muenchen.de

Helmholtz Zentrum München
Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir´in Bärbel Brumme-Bothe
Geschäftsführer: Prof. Dr. Günther Wess Dr. Nikolaus Blum Dr. Alfons Enhsen
Registergericht: Amtsgericht München HRB 6466
USt-IdNr: DE 129521671