[Spacewalk-list] c3p0 pool connection problems after 1.6 upgrade

[Roderick Grau]

Spacewalk List,

I recently updated my SW server from 1.1 incrementally to 1.6.  Restarting and checking the installation after each upgrade.  All went well until I reach the the 1.5 -> 1.6 upgrade.  Tomcat failed to start with the "Waiting for tomcat to be ready" message.

I tried to do upgrade the 1.6 install to 1.7 and still get the same error.

I have removed the 1.7 install and reinstalled 1.7 with the same error.

I checked and spacewalk user can connect to the oracle db. 

I checked that there are connections from localhost to the db on port 1521 with netstat and inside the oracle db.

I have googled for the errors and turned up nothing.

My environment:
RHEL5.6
Oracle 11g
Both on the same machine.

>From catalina.out

Using CATALINA_BASE:   /usr/share/tomcat5
Using CATALINA_HOME:   /usr/share/tomcat5
Using CATALINA_TMPDIR: /usr/share/tomcat5/temp
Using JRE_HOME:       
Mar 27, 2012 8:11:34 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64:/usr/lib/oracle/11.2/client64/lib:/usr/lib/oracle/11.2/client/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Mar 27, 2012 8:11:34 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080
Mar 27, 2012 8:11:34 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1754 ms
Mar 27, 2012 8:11:35 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Mar 27, 2012 8:11:35 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.23
Mar 27, 2012 8:11:35 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Mar 27, 2012 8:11:53 AM com.mchange.v2.log.MLog <clinit>
INFO: MLog clients using java 1.4+ standard logging.
Mar 27, 2012 8:11:53 AM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.0 [built 13-July-2007 10:11:26 -0400; debug? false; trace: 5]
Mar 27, 2012 8:11:54 AM com.mchange.v2.c3p0.PoolBackedDataSource getPoolManager
INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource at 1c87093 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 54cb2185 [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 54cb2185, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxIdleTime -> 300, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource at 651e67c [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 651e67c, jdbcUrl -> jdbc:oracle:oci:@orcl, properties -> {user=******, password=******, driver_proto=jdbc:oracle:oci!
} ], preferredTestQuery -> null, propertyCycle -> 300, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true, usesTraditionalReflectiveProxies -> false ], factoryClassLocation -> null, identityToken -> 1c87093, numHelperThreads -> 3 ]
Mar 27, 2012 8:21:44 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector at 7df33bb0 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Mar 27, 2012 8:21:44 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector at 7df33bb0 -- APPARENT DEADLOCK!!! Complete Status: [num_managed_threads: 3, num_active: 3; activeTasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask at 6bd3e069 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0), com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask at 233244cd (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2), 

... lots more lines ....

Mar 27, 2012 8:22:41 AM com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask run
WARNING: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask at 5e5f33b8 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30).
Mar 27, 2012 8:22:41 AM com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask run
WARNING: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask at 44f942c4 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30).


Any suggestions would be greatly appreciated.

-Rod