[Pulp-list] FW: New Pulp install - pulp-admin commands fail

Golden, Joel Joel.Golden at twtelecom.com
Mon Nov 21 15:57:05 UTC 2011


Running RHEL 5.6 with latest errata and followed the install on the URL you listed.  I started with the Pulp community repo then upgraded to the testing repo, which produced the same error.
________________________________________
From: James Slagle [jslagle at redhat.com]
Sent: Friday, November 18, 2011 2:43 PM
To: Golden, Joel
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] New Pulp install - pulp-admin commands fail

It looks like you're on RHEL 5 x86_64 based on the package manifest you showed.  Are you running a specific 5.X or just latest RHEL 5 with all updates applied?

I'm assuming you just walked through the instructions at http://pulpproject.org/ug/UGInstallation.html.

Let me know what version of RHEL 5 you're on, and we can try the setup and see if we can repro the issue.

Thanks.

On Fri, Nov 18, 2011 at 09:19:56PM +0000, Golden, Joel wrote:
> The only logs in /var/log/pulp are the db.log and client.log.  =/
>
> I corrected hostnames in the files you specified but still get the same error.
>
> ##pulp-admin -u admin -p admin user list
> error:  error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD 
> HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> 
> </head><body> <h1>Internal Server Error</h1> <p>The server encountered 
> an internal error or misconfiguration and was unable to complete your 
> request.</p> <p>Please contact the server administrator,  
> root at localhost and inform them of the time the error occurred, and 
> anything you might have done that may have caused the error.</p> 
> <p>More information about this error may be available in the server 
> error log.</p> <hr>
> <address>Apache/2.2.3 (Red Hat) Server at kstester-den Port 
> 443</address>
>
> ##hostname
> kstester-den
>
> ##/etc/pulp/admin/admin.conf:
> [server]
> host = kstester-den
> port = 443
> scheme = https
> path = /pulp/api
> interval = 240
>
> [plugins]
> plugin_dirs = /usr/lib/pulp/admin/plugins
>
> ##openssl x509 -text -in /etc/pki/tls/certs/localhost.crt:
> Certificate:
>     Data:
>         Version: 3 (0x2)
>         Serial Number: 28289 (0x6e81)
>         Signature Algorithm: sha1WithRSAEncryption
>         Issuer: C=--, ST=SomeState, L=SomeCity, O=SomeOrganization, OU=SomeOrganizationalUnit, CN=kstester-den/emailAddress=root at kstester-den
>         Validity
>             Not Before: Sep 23 14:01:07 2011 GMT
>             Not After : Sep 22 14:01:07 2012 GMT
>         Subject: C=--, ST=SomeState, L=SomeCity, O=SomeOrganization, 
> OU=SomeOrganizationalUnit, 
> CN=kstester-den/emailAddress=root at kstester-den
>
> ##/etc/sysconfig/network:
> NETWORKING=yes
> NETWORKING_IPV6=no
> HOSTNAME=kstester-den
>
> ##/var/log/httpd/ssl_error_log:
> [Fri Nov 18 14:12:38 2011] [info] Connection: Client IP: 10.47.28.29, 
> Protocol: TLSv1, Cipher: DHE-RSA-AES256-SHA (256/256 bits) [Fri Nov 18 
> 14:14:38 2011] [error] [client 10.47.28.29] Premature end of script 
> headers: webservices.wsgi [Fri Nov 18 14:14:38 2011] [debug] 
> ssl_engine_kernel.c(1819): OpenSSL: Write: SSL negotiation finished 
> successfully [Fri Nov 18 14:14:38 2011] [info] [client 10.47.28.29] 
> Connection closed to child 5 with standard shutdown (server 
> kstester-den:443)
>
>
> -----Original Message-----
> From: James Slagle [mailto:jslagle at redhat.com]
> Sent: Friday, November 18, 2011 1:48 PM
> To: Golden, Joel
> Cc: pulp-list at redhat.com
> Subject: Re: [Pulp-list] New Pulp install - pulp-admin commands fail
>
> Hello,
>
> I'm not sure I have anything definitive, but I did notice one thing that might be an issue.
>
> From ssl_error_log, I see:
> > [Fri Nov 18 12:35:35 2011] [info] [client 10.47.28.29] Connection 
> > closed to child 1 with standard shutdown (server
> > kstester-den.domain.com:443)
>
> I'm wondering if apache thinks your hostname is kstester-den.domain.com, but pulp is configured to expect kstester-den.
>
> What's does the command 'hostname' say on the box?  Also, what's configured in /etc/sysconfig/network?  That value needs to match what's set for server_url in /etc/pulp/pulp.conf and for host in /etc/pulp/admin/admin.conf.
>
> Also, it needs to match the CN of the subject that is set in the SSL certificate that is referenced by the SSLCertificateFile config in /etc/httpd/conf.d/ssl.conf.  If you haven't changed any default values in that file, the certificate should be /etc/pki/tls/certs/localhost.crt.
>
> You can see the certificate in text format by using:
> openssl x509 -text -in /etc/pki/tls/certs/localhost.crt
>
> You could try sending the full log files, there might be something else there.
> Do you have a /var/log/pulp/pulp.log?  I didn't see that shown below.
>
>
> On Fri, Nov 18, 2011 at 07:59:12PM +0000, Golden, Joel wrote:
> > Spent the past 2.5 days researching on a fix but no luck...  I do 
> > not want to feel defeated at the end of a Friday.  LOL
> >
> > I am experiencing a similar error to 
> > https://www.redhat.com/archives/pulp-list/2011-January/msg00049.html
> > but does not appear to be mongodb related, which is where this thread seems to stop.
> >
> > All daemons appear to come up without issue and I've searched high and low for a resolution.  Please let me know if you need additional logs or config files.  Any help will be greatly appreciated!
> >
> > pulp-admin -u admin -p admin user list
> > error:  error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD 
> > HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> 
> > </head><body> <h1>Internal Server Error</h1> <p>The server 
> > encountered an internal error or misconfiguration and was unable to 
> > complete your request.</p> <p>Please contact the server 
> > administrator, root at localhost and inform them of the time the error 
> > occurred, and anything you might have done that may have caused the 
> > error.</p> <p>More information about this error may be available in 
> > the server error log.</p> <hr>
> > <address>Apache/2.2.3 (Red Hat) Server at host.domain.com Port 
> > 443</address>
> >
> > netstat:
> > tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      31153/qpidd
> > tcp        0      0 0.0.0.0:27017               0.0.0.0:*                   LISTEN      31143/mongod
> > tcp        0      0 0.0.0.0:28017               0.0.0.0:*                   LISTEN      31143/mongod
> > tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      31192/httpd
> > tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      31192/httpd
> > tcp        0      0 10.1.23.14:43378            10.1.18.28:636              ESTABLISHED 31193/(wsgi:pulp)
> >
> > /var/log/messages:
> > Nov 18 12:33:00 kstester-den qpidd[31153]: 2011-11-18 12:33:00 
> > notice Journal "TplStore": Created Nov 18 12:33:00 kstester-den qpidd[31153]:
> > 2011-11-18 12:33:00 notice Store module initialized; 
> > store-dir=/var/lib/qpidd Nov 18 12:33:00 kstester-den qpidd[31153]:
> > 2011-11-18 12:33:00 notice SASL disabled: No Authentication 
> > Performed Nov 18 12:33:00 kstester-den qpidd[31153]: 2011-11-18 
> > 12:33:00 notice Listening on TCP port 5672 Nov 18 12:33:00 kstester-den qpidd[31153]:
> > 2011-11-18 12:33:00 notice Broker running
> >
> > /var/log/httpd/ssl_error.log:
> > [Fri Nov 18 12:33:35 2011] [debug] ssl_engine_kernel.c(1809): OpenSSL:
> > Loop: SSLv3 read finished A [Fri Nov 18 12:33:35 2011] [debug]
> > ssl_engine_kernel.c(1809): OpenSSL: Loop: SSLv3 write change cipher 
> > spec A [Fri Nov 18 12:33:35 2011] [debug] ssl_engine_kernel.c(1809):
> > OpenSSL: Loop: SSLv3 write finished A [Fri Nov 18 12:33:35 2011] 
> > [debug] ssl_engine_kernel.c(1809): OpenSSL: Loop: SSLv3 flush data 
> > [Fri Nov 18 12:33:35 2011] [debug] ssl_scache_shmcb.c(670): inside 
> > shmcb_store_session [Fri Nov 18 12:33:35 2011] [debug]
> > ssl_scache_shmcb.c(676): session_id[0]=227, masked index=3 [Fri Nov 
> > 18
> > 12:33:35 2011] [debug] ssl_scache_shmcb.c(1059): entering 
> > shmcb_insert_encoded_session, *queue->pos_count = 0 [Fri Nov 18
> > 12:33:35 2011] [debug] ssl_scache_shmcb.c(983): entering 
> > shmcb_expire_division [Fri Nov 18 12:33:35 2011] [debug]
> > ssl_scache_shmcb.c(1115): we have 13853 bytes and 133 indexes free - 
> > enough [Fri Nov 18 12:33:35 2011] [debug] ssl_scache_shmcb.c(1144):
> > storing in index 0, at offset 0 [Fri Nov 18 12:33:35 2011] [debug]
> > ssl_scache_shmcb.c(1159): session_id[0]=227, idx->s_id2=254 [Fri Nov
> > 18 12:33:35 2011] [debug] ssl_scache_shmcb.c(1170): leaving now with
> > 123 bytes in the cache and 1 indexes [Fri Nov 18 12:33:35 2011] 
> > [debug] ssl_scache_shmcb.c(1174): leaving 
> > shmcb_insert_encoded_session [Fri Nov 18 12:33:35 2011] [debug] 
> > ssl_scache_shmcb.c(704): leaving shmcb_store successfully [Fri Nov 
> > 18 12:33:35 2011] [debug]
> > ssl_scache_shmcb.c(418): shmcb_store successful [Fri Nov 18 12:33:35 
> > 2011] [debug] ssl_engine_kernel.c(1667): Inter-Process Session Cache:
> > request=SET status=OK
> > id=E3FE4F919F2FDADFBF497D237089584DD60494E2F9430CFB58D5651005E77C71
> > timeout=300s (session caching) [Fri Nov 18 12:33:35 2011] [debug]
> > ssl_engine_kernel.c(1805): OpenSSL: Handshake: done [Fri Nov 18
> > 12:33:35 2011] [info] Connection: Client IP: 10.47.28.29, Protocol:
> > TLSv1, Cipher: DHE-RSA-AES256-SHA (256/256 bits) [Fri Nov 18 
> > 12:35:35 2011] [error] [client 10.47.28.29] Premature end of script headers:
> > webservices.wsgi [Fri Nov 18 12:35:35 2011] [debug]
> > ssl_engine_kernel.c(1819): OpenSSL: Write: SSL negotiation finished 
> > successfully [Fri Nov 18 12:35:35 2011] [info] [client 10.47.28.29] 
> > Connection closed to child 1 with standard shutdown (server
> > kstester-den.domain.com:443)
> >
> > /var/log/httpd/ssl_access_log:
> > 10.47.28.29 - - [18/Nov/2011:11:40:48 -0700] "GET /pulp/api//users/ 
> > HTTP/1.1" 500 627
> > 10.47.28.29 - - [18/Nov/2011:11:50:24 -0700] "GET /pulp/api//users/ 
> > HTTP/1.1" 500 627
> > 10.47.28.29 - - [18/Nov/2011:12:18:07 -0700] "GET /pulp/api//users/ 
> > HTTP/1.1" 500 627
> > 10.47.28.29 - - [18/Nov/2011:12:27:33 -0700] "GET /pulp/api//users/ 
> > HTTP/1.1" 500 627
> > 10.47.28.29 - - [18/Nov/2011:12:33:35 -0700] "GET /pulp/api//users/ 
> > HTTP/1.1" 500 627
> >
> > /var/log/httpd/ssl_request_log:
> > [18/Nov/2011:11:40:48 -0700] 10.47.28.29 TLSv1 DHE-RSA-AES256-SHA 
> > "GET /pulp/api//users/ HTTP/1.1" 627
> > [18/Nov/2011:11:50:24 -0700] 10.47.28.29 TLSv1 DHE-RSA-AES256-SHA 
> > "GET /pulp/api//users/ HTTP/1.1" 627
> > [18/Nov/2011:12:18:07 -0700] 10.47.28.29 TLSv1 DHE-RSA-AES256-SHA 
> > "GET /pulp/api//users/ HTTP/1.1" 627
> > [18/Nov/2011:12:27:33 -0700] 10.47.28.29 TLSv1 DHE-RSA-AES256-SHA 
> > "GET /pulp/api//users/ HTTP/1.1" 627
> > [18/Nov/2011:12:33:35 -0700] 10.47.28.29 TLSv1 DHE-RSA-AES256-SHA 
> > "GET /pulp/api//users/ HTTP/1.1" 627
> >
> > /var/log/httpd/error_log:
> > [Fri Nov 18 12:33:01 2011] [info] mod_wsgi (pid=31194): Attach interpreter ''.
> > [Fri Nov 18 12:33:01 2011] [debug] proxy_util.c(1817): proxy: 
> > grabbed scoreboard slot 0 in child 31202 for worker proxy:reverse 
> > [Fri Nov 18
> > 12:33:01 2011] [debug] proxy_util.c(1836): proxy: worker 
> > proxy:reverse already initialized [Fri Nov 18 12:33:01 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 31202 for (*) [Fri Nov 18 12:33:01 2011] [info] mod_wsgi (pid=31202): Attach interpreter ''.
> > [Fri Nov 18 12:33:01 2011] [debug] proxy_util.c(1817): proxy: 
> > grabbed scoreboard slot 0 in child 31200 for worker proxy:reverse 
> > [Fri Nov 18
> > 12:33:01 2011] [debug] proxy_util.c(1836): proxy: worker 
> > proxy:reverse already initialized [Fri Nov 18 12:33:01 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 31200 for (*) [Fri Nov 18 12:33:01 2011] [info] mod_wsgi (pid=31200): Attach interpreter ''.
> >
> > /var/log/mongodb/mongodb.log:
> > Fri Nov 18 12:33:00 [initandlisten] Accessing: pulp_database for the 
> > first time Fri Nov 18 12:33:00 [initandlisten] done repairDatabases 
> > Fri Nov 18 12:33:00 [initandlisten] waiting for connections on port
> > 27017 Fri Nov 18 12:33:00 [websvr] web admin interface listening on 
> > port 28017 Fri Nov 18 12:33:00 [websvr] fd limit hard:1024 soft:1024 
> > max conn: 819 Fri Nov 18 12:33:00 [initandlisten] fd limit hard:1024
> > soft:1024 max conn: 819 Fri Nov 18 12:34:00 [DataFileSync] flushing 
> > mmap took 0ms  for 1 files Fri Nov 18 12:35:00 [DataFileSync] 
> > flushing mmap took 0ms  for 1 files
> >
> > /var/log/pulp/db.log:
> > validation on previous data model version started validation 
> > complete; data model at version 25 migration to data model version 
> > 26 started migration to data model version 26 complete migration to 
> > data model version 27 started migration to data model version 27 
> > complete migration to data model version 28 started migration to 
> > data model version 28 complete **M2Crypto<%s> lacks patch for using 
> > Certificate Revocation Lists** migration to data model version 29 
> > started migration to data model version 29 complete
> >
> > /var/log/pulp/client.log:
> > 2011-11-18 12:35:35,293 [ERROR][MainThread] @ command.py:226 - error:
> > (500, '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 
> > 2.0//EN">\n<html><head>\n<title>500 Internal Server 
> > Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The 
> > server encountered an internal error or\nmisconfiguration and was 
> > unable to complete\nyour request.</p>\n<p>Please contact the server 
> > administrator,\n root at localhost and inform them of the time the 
> > error occurred,\nand anything you might have done that may 
> > have\ncaused the error.</p>\n<p>More information about this error 
> > may be available\nin the server error 
> > log.</p>\n<hr>\n<address>Apache/2.2.3 (Red Hat) Server at 
> > kstester-den.twtelecom.com Port 443</address>\n</body></html>\n', 
> > None)
> >
> >
> > /etc/pulp.conf:
> > [server]
> > server_name: kstester-den
> > relative_url: /pulp/repos
> > key_url: /pulp/gpg
> > ks_url: /pulp/ks
> > default_login: admin
> > default_password: admin
> > debugging_mode: true
> >
> > [messaging]
> > url:        tcp://localhost:5672
> > cacert:     /etc/pki/qpid/ca/ca.crt
> > clientcert: /etc/pki/qpid/client/client.pem
> >
> > [events]
> > send_enabled: false
> > recv_enabled: false
> >
> > [security]
> > cacert:  /etc/pki/pulp/ca.crt
> > cakey:   /etc/pki/pulp/ca.key
> > user_cert_expiration: 7
> > consumer_cert_expiration: 3650
> >
> > [auditing]
> > audit_events: true
> > events_file: /var/log/pulp/events.log
> > lifetime:    90
> > backups:     4
> >
> > [consumer_history]
> > lifetime: 180
> >
> > [logs]
> > config: /etc/pulp/logging/debug_sync.cfg
> >
> > [yum]
> > threads: 15
> > remove_old_versions: false
> > num_old_pkgs_keep: 2
> > verify_checksum: true
> > verify_size: true
> >
> > [repos]
> > content_url: https://cdn.redhat.com/ use_entitlement_certs = false
> > default_to_published: true
> >
> > [database]
> > name: pulp_database
> > seeds: localhost
> >
> > [cds]
> > sync_timeout = 10:7200
> >
> >
> >
> > Version info:
> > pulp-client-lib-0.0.249-1
> > pulp-admin-0.0.249-1
> > mod_wsgi-3.2-6.pulp
> > pulp-common-0.0.249-1
> > pulp-0.0.249-1
> > mongodb-server-1.6.4-1.el5
> > pymongo-1.9-8.el5
> > mongodb-1.6.4-1.el5
> > qpid-cpp-client-0.7.946106-34.el5
> > qpid-cpp-server-store-0.7.946106-34.el5
> > qpid-cpp-server-0.7.946106-34.el5
> > python-qpid-0.7.946106-15.el5
> > qpid-cpp-server-ssl-0.7.946106-34.el5
> > qpid-cpp-client-ssl-0.7.946106-34.el5
> > amqp-1.0.750054-1.el5.noarch
> > python-ssl-1.15-3.el5.x86_64
> > xz-libs-4.999.9-0.3.beta.20091007git.el5.x86_64
> > python-hashlib-20081119-4.el5.x86_64
> > js-1.70-8.el5.x86_64
> > mod_wsgi-3.2-6.pulp.x86_64
> > python-pycurl-7.15.5.1-8.el5.x86_64
> > gofer-0.59-1.noarch
> > grinder-0.0.129-1.noarch
> > mod_python-3.2.8-3.1
> > python-2.4.3-44.el5
> > httpd-2.2.3-53.el5_7.3
> > python-httplib2-0.4.0-4.el5
> >
> > Joel Golden
> >
> >
> > -------------
> >
> >
> >
> > The content contained in this electronic message is not intended to constitute formation of a contract binding tw telecom. tw telecom will be contractually bound only upon execution, by an authorized officer, of a contract including agreed terms and conditions or by express application of its tariffs. This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail or by telephone.
> >
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> --
> -- James Slagle
> --
>
>
--
-- James Slagle
--






More information about the Pulp-list mailing list