[Spacewalk-list] Problem with new satellite server deployment

Wed Dec 24 00:06:07 UTC 2014

Thanks Paul.  I downloaded the jar file from postgresql's site, as per
your recommendation and updated the links under /usr/share/java and
restarted spacewalk-services.  I no longer see the errors in
rhn_taskomatic_daemon.log file but the client still shows 0 packages
in those 2 channels.  Is there something I need to do in order for the
repodata to be generated?  Am I missing a step?  Would it be
recommended to use a external postgres db or a embedded one?

Thanks for all your help with this.  It is much appreciated!

On Tue, Dec 23, 2014 at 3:30 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
> Yes absolutely I've hit this too.
>
> You need to upgrade the jdbc driver on the box running spacewalk to match
> the version of PostgreSQL. You can download it off the PostgreSQL web site.
> Its a jar file you need to download it to the correct directory then modify
> the symlinks to point to it.
>
> I've written instructions on how to do this in the past on this list if you
> can find it in google that's great. But if you can't let me know and I'll
> email the instructions when I get back in the office tomorrow morning.
>
>
>
> -- Sent from my HP Pre3
>
> ________________________________
> On Dec 23, 2014 6:02 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>
> Thanks for your reply Paul. Looks like my dab's are running version 9.3:
>
> /opt/postgres/pgserver9.3/bin/postgres
>
> You think this is related to the db side?
>
>
> On Tue, Dec 23, 2014 at 2:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>> Since I am not using this in production yet, I decided to upgrade
>> everything to the latest available on the spacewalk server running
>> CentOS. Did not touch the external postgres server running on RHEL.
>> However, still getting the same issue. This is the complete error
>> when in the rhn_taskomatic_daemon.log file when restarting spacewalk:
>>
>> --
>> STATUS | wrapper | 2014/12/23 14:09:31 | TERM trapped. Shutting down.
>> STATUS | wrapper | 2014/12/23 14:09:32 | <-- Wrapper Stopped
>> STATUS | wrapper | 2014/12/23 14:10:08 | --> Wrapper Started as Daemon
>> STATUS | wrapper | 2014/12/23 14:10:08 | Launching a JVM...
>> INFO | jvm 1 | 2014/12/23 14:10:08 | Wrapper (Version 3.2.3)
>> http://wrapper.tanukisoftware.org
>> INFO | jvm 1 | 2014/12/23 14:10:08 | Copyright 1999-2006 Tanuki
>> Software, Inc. All Rights Reserved.
>> INFO | jvm 1 | 2014/12/23 14:10:08 |
>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>> com.mchange.v2.log.MLog <clinit>
>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: MLog clients using
>> java 1.4+ standard logging.
>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>> com.mchange.v2.c3p0.C3P0Registry banner
>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing
>> c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5]
>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing c3p0
>> pool... com.mchange.v2.c3p0.PoolBackedDataSource at 585a1c43 [
>> connectionPoolDataSource ->
>> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 1c136579 [
>> acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
>> -> 1000, autoCommitOnClose -> false, automaticTestTable -> null,
>> breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
>> connectionCustomizerClassName ->
>> com.redhat.rhn.common.db.RhnConnectionCustomizer,
>> connectionTesterClassName ->
>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>> debugUnreturnedConnectionStackTraces -> false, factoryClassLocation ->
>> null, forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> k0jcdo96xmmxcv1leoshp|2d09bbf2, idleConnectionTestPeriod -> 300,
>> initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge
>> -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0,
>> maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection ->
>> 0, minPoolSize -> 5, nestedDataSource ->
>> com.mchange.v2.c3p0.DriverManagerDataSource at f7c08283 [ description ->
>> null, driverClass -> null, factoryClassLocation -> null, identityToken
>> -> k0jcdo96xmmxcv1leoshp|7451e191, jdbcUrl ->
>> jdbc:postgresql://rwcswalkpg01:5432/spaceschema, properties ->
>> {user=******, password=******, driver_proto=jdbc:postgresql} ],
>> preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle ->
>> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true,
>> unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies ->
>> false; userOverrides: {} ], dataSourceName -> null,
>> factoryClassLocation -> null, identityToken ->
>> k0jcdo96xmmxcv1leoshp|2fe19787, numHelperThreads -> 3 ]
>> FATAL | jvm 1 | 2014/12/23 14:10:16 | Failure occured during job recovery.
>> com.redhat.rhn.taskomatic.core.TaskomaticException: Failure occured
>> during job recovery.
>> at
>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:158)
>> at
>> com.redhat.rhn.taskomatic.core.TaskomaticDaemon$1.run(TaskomaticDaemon.java:102)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.quartz.SchedulerConfigException: Failure occured during
>> job recovery. [See nested exception:
>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>> nested exception: java.lang.ArrayIndexOutOfBoundsException: 2]]
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:627)
>> at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:494)
>> at org.quartz.impl.StdScheduler.start(StdScheduler.java:143)
>> at
>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:146)
>> ... 2 more
>> Caused by: org.quartz.JobPersistenceException: Couldn't retrieve
>> trigger: 2 [See nested exception:
>> java.lang.ArrayIndexOutOfBoundsException: 2]
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:834)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$2.execute(JobStoreSupport.java:806)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3729)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3725)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:802)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:625)
>> ... 5 more
>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
>> at org.postgresql.util.PGbytea.toBytes(PGbytea.java:74)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2253)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
>> at
>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>> at
>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>> at
>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>> at
>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>> ... 13 more
>> --
>>
>> Need help with troubleshooting this because I can't seem to get any
>> further.
>>
>> Thanks!
>>
>> On Tue, Dec 23, 2014 at 1:25 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>> Hi Guys,
>>>
>>> I am setting up a new satellite server with a external postgres db and
>>> just
>>> cloned my first CentOS base/extras channel. On the client side everything
>>> seems to have gone smoothly except when I now run "yum repolist" I see 0
>>> packages for these 2 channels even though there are packages in those
>>> channels. One thing I have noticed is that I am seeing tons of the
>>> following errors in my /var/log/rhn/rhn_taskomatic_daemon.log log file on
>>> the spacewalk server:
>>>
>>> --
>>> ..snip..
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | 2014-12-23 13:16:23,458
>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>> ERROR
>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>> misfires: Couldn't retrieve trigger: 2
>>> INFO | jvm 1 | 2014/12/23 13:16:23 |
>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>> nested
>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | Caused by:
>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>> INFO | jvm 1 | 2014/12/23 13:16:23 | ... 4 more
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | 2014-12-23 13:20:23,458
>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>> ERROR
>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>> misfires: Couldn't retrieve trigger: 2
>>> INFO | jvm 1 | 2014/12/23 13:20:23 |
>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>> nested
>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | Caused by:
>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>> INFO | jvm 1 | 2014/12/23 13:20:23 | ... 4 more
>>> ..snip..
>>> --
>>>
>>> Any help would be greatly appreciated.
>>>
>>> Thanks!
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list