[Spacewalk-list] Problem with new satellite server deployment

Jagga Soorma jagga13 at gmail.com
Tue Dec 23 22:56:53 UTC 2014


Thanks for your reply Paul.  Looks like my dab's are running version 9.3:

/opt/postgres/pgserver9.3/bin/postgres

You think this is related to the db side?


On Tue, Dec 23, 2014 at 2:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
> Since I am not using this in production yet, I decided to upgrade
> everything to the latest available on the spacewalk server running
> CentOS.  Did not touch the external postgres server running on RHEL.
> However, still getting the same issue.  This is the complete error
> when in the rhn_taskomatic_daemon.log file when restarting spacewalk:
>
> --
> STATUS | wrapper  | 2014/12/23 14:09:31 | TERM trapped.  Shutting down.
> STATUS | wrapper  | 2014/12/23 14:09:32 | <-- Wrapper Stopped
> STATUS | wrapper  | 2014/12/23 14:10:08 | --> Wrapper Started as Daemon
> STATUS | wrapper  | 2014/12/23 14:10:08 | Launching a JVM...
> INFO   | jvm 1    | 2014/12/23 14:10:08 | Wrapper (Version 3.2.3)
> http://wrapper.tanukisoftware.org
> INFO   | jvm 1    | 2014/12/23 14:10:08 |   Copyright 1999-2006 Tanuki
> Software, Inc.  All Rights Reserved.
> INFO   | jvm 1    | 2014/12/23 14:10:08 |
> INFO   | jvm 1    | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
> com.mchange.v2.log.MLog <clinit>
> INFO   | jvm 1    | 2014/12/23 14:10:12 | INFO: MLog clients using
> java 1.4+ standard logging.
> INFO   | jvm 1    | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
> com.mchange.v2.c3p0.C3P0Registry banner
> INFO   | jvm 1    | 2014/12/23 14:10:12 | INFO: Initializing
> c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5]
> INFO   | jvm 1    | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
> INFO   | jvm 1    | 2014/12/23 14:10:12 | INFO: Initializing c3p0
> pool... com.mchange.v2.c3p0.PoolBackedDataSource at 585a1c43 [
> connectionPoolDataSource ->
> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 1c136579 [
> acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
> -> 1000, autoCommitOnClose -> false, automaticTestTable -> null,
> breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
> connectionCustomizerClassName ->
> com.redhat.rhn.common.db.RhnConnectionCustomizer,
> connectionTesterClassName ->
> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> debugUnreturnedConnectionStackTraces -> false, factoryClassLocation ->
> null, forceIgnoreUnresolvedTransactions -> false, identityToken ->
> k0jcdo96xmmxcv1leoshp|2d09bbf2, idleConnectionTestPeriod -> 300,
> initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge
> -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0,
> maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection ->
> 0, minPoolSize -> 5, nestedDataSource ->
> com.mchange.v2.c3p0.DriverManagerDataSource at f7c08283 [ description ->
> null, driverClass -> null, factoryClassLocation -> null, identityToken
> -> k0jcdo96xmmxcv1leoshp|7451e191, jdbcUrl ->
> jdbc:postgresql://rwcswalkpg01:5432/spaceschema, properties ->
> {user=******, password=******, driver_proto=jdbc:postgresql} ],
> preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle ->
> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true,
> unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies ->
> false; userOverrides: {} ], dataSourceName -> null,
> factoryClassLocation -> null, identityToken ->
> k0jcdo96xmmxcv1leoshp|2fe19787, numHelperThreads -> 3 ]
> FATAL  | jvm 1    | 2014/12/23 14:10:16 | Failure occured during job recovery.
> com.redhat.rhn.taskomatic.core.TaskomaticException: Failure occured
> during job recovery.
> at com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:158)
> at com.redhat.rhn.taskomatic.core.TaskomaticDaemon$1.run(TaskomaticDaemon.java:102)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.quartz.SchedulerConfigException: Failure occured during
> job recovery. [See nested exception:
> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
> nested exception: java.lang.ArrayIndexOutOfBoundsException: 2]]
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:627)
> at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:494)
> at org.quartz.impl.StdScheduler.start(StdScheduler.java:143)
> at com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:146)
> ... 2 more
> Caused by: org.quartz.JobPersistenceException: Couldn't retrieve
> trigger: 2 [See nested exception:
> java.lang.ArrayIndexOutOfBoundsException: 2]
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:834)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport$2.execute(JobStoreSupport.java:806)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3729)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3725)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:802)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:625)
> ... 5 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
> at org.postgresql.util.PGbytea.toBytes(PGbytea.java:74)
> at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2253)
> at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
> at org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
> at org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
> at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
> at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
> ... 13 more
> --
>
> Need help with troubleshooting this because I can't seem to get any further.
>
> Thanks!
>
> On Tue, Dec 23, 2014 at 1:25 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>> Hi Guys,
>>
>> I am setting up a new satellite server with a external postgres db and just
>> cloned my first CentOS base/extras channel.  On the client side everything
>> seems to have gone smoothly except when I now run "yum repolist" I see 0
>> packages for these 2 channels even though there are packages in those
>> channels.  One thing I have noticed is that I am seeing tons of the
>> following errors in my /var/log/rhn/rhn_taskomatic_daemon.log log file on
>> the spacewalk server:
>>
>> --
>> ..snip..
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | 2014-12-23 13:16:23,458
>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler] ERROR
>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>> misfires: Couldn't retrieve trigger: 2
>> INFO   | jvm 1    | 2014/12/23 13:16:23 |
>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See nested
>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | Caused by:
>> java.lang.ArrayIndexOutOfBoundsException: 2
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>> INFO   | jvm 1    | 2014/12/23 13:16:23 | ... 4 more
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | 2014-12-23 13:20:23,458
>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler] ERROR
>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>> misfires: Couldn't retrieve trigger: 2
>> INFO   | jvm 1    | 2014/12/23 13:20:23 |
>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See nested
>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | Caused by:
>> java.lang.ArrayIndexOutOfBoundsException: 2
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>> INFO   | jvm 1    | 2014/12/23 13:20:23 | ... 4 more
>> ..snip..
>> --
>>
>> Any help would be greatly appreciated.
>>
>> Thanks!




More information about the Spacewalk-list mailing list