<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="EN-CA" link="blue" vlink="purple"> <div class="WordSection1"> So if anyone else is running into this issue.<o:p></o:p> <o:p> </o:p> I updated the concurrent connections allowed in oracle to 200 which appears to have so far resolved the issue. If anyone else runs into this give this a try seems to do the trick<o:p></o:p> <o:p> </o:p> http://my.bergersoft.net/2010/01/06/how-to-resolve-the-oracle-max-connection-limit-exceeded-problem-on-oracle-10g-express/<o:p></o:p> <o:p> </o:p> <div style="mso-element:para-border-div;border:none;border-left:solid #AAAAAA 3.0pt;padding:0cm 0cm 0cm 12.0pt;background:#FEFEFE;margin-left:12.0pt;margin-right:0cm"> sqlplus 'sys/password@//localhost/XE as sysdba'<o:p></o:p> show parameter process;<o:p></o:p> SQL> show parameter process;<o:p></o:p> <o:p> </o:p> NAME TYPE VALUE<o:p></o:p> ------------------------------------ ----------- ------------------------------<o:p></o:p> aq_tm_processes integer 0<o:p></o:p> cell_offload_processing boolean TRUE<o:p></o:p> db_writer_processes integer 1<o:p></o:p> gcs_server_processes integer 0<o:p></o:p> global_txn_processes integer 1<o:p></o:p> job_queue_processes integer 4<o:p></o:p> log_archive_max_processes integer 4<o:p></o:p> processes integer 100<o:p></o:p> <o:p> </o:p> alter system set processes = 200 scope=spfile;<o:p></o:p> restart oracle server<o:p></o:p> <o:p> </o:p> Check the process<o:p></o:p> SQL> show parameter process;<o:p></o:p> <o:p> </o:p> NAME TYPE VALUE<o:p></o:p> ------------------------------------ ----------- ------------------------------<o:p></o:p> aq_tm_processes integer 0<o:p></o:p> cell_offload_processing boolean TRUE<o:p></o:p> db_writer_processes integer 1<o:p></o:p> gcs_server_processes integer 0<o:p></o:p> global_txn_processes integer 1<o:p></o:p> job_queue_processes integer 4<o:p></o:p> log_archive_max_processes integer 4<o:p></o:p> processes integer 200<o:p></o:p> </div> <o:p> </o:p> <o:p> </o:p> Nick Tailor<o:p></o:p> Nicktailor.com<o:p></o:p> <o:p> </o:p> <o:p> </o:p> From: spacewalk-list-bounces@redhat.com [mailto:spacewalk-list-bounces@redhat.com] On Behalf Of Waldirio Manhães Pinheiro Sent: December-16-14 10:43 AM To: spacewalk-list@redhat.com Subject: Re: [Spacewalk-list] rhnpush intermittent 500 internal error<o:p></o:p> <o:p> </o:p> <div> Nick<o:p></o:p> <div> <o:p> </o:p> </div> <div> Recommend you open a new BZ about it and continue with your tests, maybe the developer team will fix asap.<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> B'Regards and congratz!<o:p></o:p> </div> </div> <div> <o:p></o:p> <div> <div> <div> ______________ Atenciosamente Waldirio msn: <a href="mailto:waldirio@gmail.com" target="_blank">waldirio@gmail.com</a> Skype: waldirio Site: <a href="http://www.waldirio.com.br" target="_blank">www.waldirio.com.br</a> Blog: <a href="http://blog.waldirio.com.br" target="_blank">blog.waldirio.com.br</a><o:p></o:p> </div> <div> LinkedIn: <a href="http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646" target="_blank"> http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646</a> PGP: <a href="http://www.waldirio.com.br/public.html" target="_blank">www.waldirio.com.br/public.html</a><o:p></o:p> </div> </div> </div> <o:p> </o:p> <div> On Tue, Dec 16, 2014 at 4:30 PM, Nick Tailor <<a href="mailto:HTailor@bclc.com" target="_blank">HTailor@bclc.com</a>> wrote:<o:p></o:p> <div> <div> The issue appears to be with oracle a quick restart of it solved the issue.<o:p></o:p> So the rhnpush is opening too many concurrent connections which are not closing on the oracle side, so after it hits a certain number it stops listening, similar to that of apache.<o:p></o:p> So will need to either increase the concurrent connection or ensure the rhnpush closes the connections after each push.<o:p></o:p> Cheers<o:p></o:p> <div> Nick Tailor<o:p></o:p> Nicktailor.com<o:p></o:p> <o:p></o:p> </div> <o:p></o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Nick Tailor Sent: December-16-14 10:27 AM To: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> Subject: RE: rhnpush intermittent 500 internal error<o:p></o:p> </div> </div> <div> <div> <o:p></o:p> The issue appears to be with oracle. <o:p></o:p> <o:p></o:p> <o:p></o:p> <div> Nick Tailor<o:p></o:p> Nicktalor.com<o:p></o:p> <o:p></o:p> </div> <o:p></o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Nick Tailor Sent: December-16-14 10:11 AM To: <a href="mailto:spacewalk-list@redhat.com" target="_blank">spacewalk-list@redhat.com</a> Subject: rhnpush intermittent 500 internal error<o:p></o:p> </div> </div> <o:p></o:p> Has anyone else seen this?<o:p></o:p> <o:p></o:p> I found someone else who had the exact same problem, however no one seemed to have replied to its resolution?<o:p></o:p> <o:p></o:p> <a href="http://www.redhat.com/archives/spacewalk-list/2013-February/msg00068.html" target="_blank">http://www.redhat.com/archives/spacewalk-list/2013-February/msg00068.html</a> (Similar problem)<o:p></o:p> <o:p></o:p> Has anyone experienced a problem during the rhnpush <o:p></o:p> <o:p></o:p> Internal server error 500 Internal Server Error Error pushing /root/BAC/TIVsm-JBB.x86_64.rpm: (500) Waiting 2 seconds and trying again... Uploading package /root/BAC/TIVsm-JBB.x86_64.rpm Using POST request Internal server error 500 Internal Server Error Error pushing /root/BAC/TIVsm-JBB.x86_64.rpm: (500) Waiting 1 seconds and trying again... Giving up after 3 attempts<o:p></o:p> <o:p></o:p> I checked the /var/log/rhn/rhn_upload_package_push.log<o:p></o:p> <o:p></o:p> So then I check the oracle tns listener db logs, it appears the listener intermittently for spacewalk stops listening which causes the rhnpush to fail, and return a internal 500 error. I have updated the hosts file and changed the rhn.conf to use the ip instead of the db host name to see if that help? I would like to know if anyone else has experienced this and what they did to resolute the issue?<o:p></o:p> <o:p></o:p> Please and tahnk you<o:p></o:p> <o:p></o:p> <o:p></o:p> 2014/12/15 20:38:22 -07:00 6407 ::1: server/apacheUploadServer.handler('GET method received, returning',)<o:p></o:p> 2014/12/15 20:38:22 -07:00 6394 ::1: rhnSQL/driver_cx_Oracle.connect('Connecting to database', 'spacewalk@//<a href="http://kam100.db.com/XE" target="_blank">kam100.db.com/XE</a>')<o:p></o:p> 2014/12/15 20:38:22 -07:00 6394 ::1: rhnSQL/driver_cx_Oracle.connect('ERROR', 'Connection attempt failed', 12516, 'ORA-12516: TNS:listener could not find available handler with matching protocol stack\n')<o:p></o:p> 2014/12/15 20:38:22 -07:00 6394 ::1: server/apacheUploadServer._wrapper('ERROR', 'Unhandled exception')<o:p></o:p> 2014/12/15 20:38:26 -07:00 5510 ::1: package_push/package_push.headerParserHandler('Username', None, 'Force', 0)<o:p></o:p> <o:p></o:p> <o:p></o:p> /listener/alert/log.xml<o:p></o:p> <msg time='2014-12-16T09:48:07.721-08:00' org_id='oracle' comp_id='tnslsnr'<o:p></o:p> type='UNKNOWN' level='16' host_id= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a><o:p></o:p> host_addr='10.0.9.107'><o:p></o:p> <txt>16-DEC-2014 09:48:07 * (CONNECT_DATA=(SERVICE_NAME=XE)(CID=(PROGRAM=java)(HOST= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a>)(USER=tomcat))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.9.101)(PORT=45479)) * establish * XE * 12516<o:p></o:p> </txt><o:p></o:p> </msg><o:p></o:p> <msg time='2014-12-16T09:48:07.721-08:00' org_id='oracle' comp_id='tnslsnr'<o:p></o:p> type='UNKNOWN' level='16' host_id= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a> host_addr='10.0.9.107'><o:p></o:p> <txt>TNS-12516: TNS:listener could not find available handler with matching protocol stack<o:p></o:p> </txt><o:p></o:p> </msg><o:p></o:p> <msg time='2014-12-16T09:48:07.722-08:00' org_id='oracle' comp_id='tnslsnr'<o:p></o:p> type='UNKNOWN' level='16' host_id= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a><o:p></o:p> host_addr='10.0.9.107'><o:p></o:p> <txt>16-DEC-2014 09:48:07 * (CONNECT_DATA=(SERVICE_NAME=XE)(CID=(PROGRAM=java)(HOST= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a>)(USER=tomcat))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.9.101)(PORT=45480)) * establish * XE * 12516<o:p></o:p> </txt><o:p></o:p> </msg><o:p></o:p> <msg time='2014-12-16T09:48:07.722-08:00' org_id='oracle' comp_id='tnslsnr'<o:p></o:p> type='UNKNOWN' level='16' host_id= <a href="http://kam100.db.com" target="_blank">kam100.db.com</a><o:p></o:p> host_addr='10.0.9.107'><o:p></o:p> <txt>TNS-12516: TNS:listener could not find available handler with matching protocol stack<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> Nick Tailor<o:p></o:p> Nicktailor.com<o:p></o:p> <o:p></o:p> <o:p></o:p> </div> </div> </div> <div> <div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="100%" align="center"> </div> This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email. <o:p></o:p> </div> </div> </div> _______________________________________________ Spacewalk-list mailing list <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><o:p></o:p> </div> </div> </div> <hr> This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email. </body> </html>