[Spacewalk-list] Problem with new satellite server deployment

Wed Dec 24 00:14:16 UTC 2014

Ahh I was able to manually run the scheduled task to generate repodata
and now see that the repository metadata for my channels has been
generated successfully in the rhn_taskomatic_daemon.log file.  However
the client still shows 0 packages.  Now I am not sure what else to
look for or what I might be missing.  Also, would like to know your
thoughts about using a external postgres db or not.

Thanks much.

On Tue, Dec 23, 2014 at 4:06 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
> Thanks Paul.  I downloaded the jar file from postgresql's site, as per
> your recommendation and updated the links under /usr/share/java and
> restarted spacewalk-services.  I no longer see the errors in
> rhn_taskomatic_daemon.log file but the client still shows 0 packages
> in those 2 channels.  Is there something I need to do in order for the
> repodata to be generated?  Am I missing a step?  Would it be
> recommended to use a external postgres db or a embedded one?
>
> Thanks for all your help with this.  It is much appreciated!
>
> On Tue, Dec 23, 2014 at 3:30 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
>> Yes absolutely I've hit this too.
>>
>> You need to upgrade the jdbc driver on the box running spacewalk to match
>> the version of PostgreSQL. You can download it off the PostgreSQL web site.
>> Its a jar file you need to download it to the correct directory then modify
>> the symlinks to point to it.
>>
>> I've written instructions on how to do this in the past on this list if you
>> can find it in google that's great. But if you can't let me know and I'll
>> email the instructions when I get back in the office tomorrow morning.
>>
>>
>>
>> -- Sent from my HP Pre3
>>
>> ________________________________
>> On Dec 23, 2014 6:02 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>
>> Thanks for your reply Paul. Looks like my dab's are running version 9.3:
>>
>> /opt/postgres/pgserver9.3/bin/postgres
>>
>> You think this is related to the db side?
>>
>>
>> On Tue, Dec 23, 2014 at 2:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>> Since I am not using this in production yet, I decided to upgrade
>>> everything to the latest available on the spacewalk server running
>>> CentOS. Did not touch the external postgres server running on RHEL.
>>> However, still getting the same issue. This is the complete error
>>> when in the rhn_taskomatic_daemon.log file when restarting spacewalk:
>>>
>>> --
>>> STATUS | wrapper | 2014/12/23 14:09:31 | TERM trapped. Shutting down.
>>> STATUS | wrapper | 2014/12/23 14:09:32 | <-- Wrapper Stopped
>>> STATUS | wrapper | 2014/12/23 14:10:08 | --> Wrapper Started as Daemon
>>> STATUS | wrapper | 2014/12/23 14:10:08 | Launching a JVM...
>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Wrapper (Version 3.2.3)
>>> http://wrapper.tanukisoftware.org
>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Copyright 1999-2006 Tanuki
>>> Software, Inc. All Rights Reserved.
>>> INFO | jvm 1 | 2014/12/23 14:10:08 |
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>> com.mchange.v2.log.MLog <clinit>
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: MLog clients using
>>> java 1.4+ standard logging.
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>> com.mchange.v2.c3p0.C3P0Registry banner
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing
>>> c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5]
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing c3p0
>>> pool... com.mchange.v2.c3p0.PoolBackedDataSource at 585a1c43 [
>>> connectionPoolDataSource ->
>>> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 1c136579 [
>>> acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
>>> -> 1000, autoCommitOnClose -> false, automaticTestTable -> null,
>>> breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
>>> connectionCustomizerClassName ->
>>> com.redhat.rhn.common.db.RhnConnectionCustomizer,
>>> connectionTesterClassName ->
>>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>>> debugUnreturnedConnectionStackTraces -> false, factoryClassLocation ->
>>> null, forceIgnoreUnresolvedTransactions -> false, identityToken ->
>>> k0jcdo96xmmxcv1leoshp|2d09bbf2, idleConnectionTestPeriod -> 300,
>>> initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge
>>> -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0,
>>> maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection ->
>>> 0, minPoolSize -> 5, nestedDataSource ->
>>> com.mchange.v2.c3p0.DriverManagerDataSource at f7c08283 [ description ->
>>> null, driverClass -> null, factoryClassLocation -> null, identityToken
>>> -> k0jcdo96xmmxcv1leoshp|7451e191, jdbcUrl ->
>>> jdbc:postgresql://rwcswalkpg01:5432/spaceschema, properties ->
>>> {user=******, password=******, driver_proto=jdbc:postgresql} ],
>>> preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle ->
>>> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true,
>>> unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies ->
>>> false; userOverrides: {} ], dataSourceName -> null,
>>> factoryClassLocation -> null, identityToken ->
>>> k0jcdo96xmmxcv1leoshp|2fe19787, numHelperThreads -> 3 ]
>>> FATAL | jvm 1 | 2014/12/23 14:10:16 | Failure occured during job recovery.
>>> com.redhat.rhn.taskomatic.core.TaskomaticException: Failure occured
>>> during job recovery.
>>> at
>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:158)
>>> at
>>> com.redhat.rhn.taskomatic.core.TaskomaticDaemon$1.run(TaskomaticDaemon.java:102)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.quartz.SchedulerConfigException: Failure occured during
>>> job recovery. [See nested exception:
>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>> nested exception: java.lang.ArrayIndexOutOfBoundsException: 2]]
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:627)
>>> at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:494)
>>> at org.quartz.impl.StdScheduler.start(StdScheduler.java:143)
>>> at
>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:146)
>>> ... 2 more
>>> Caused by: org.quartz.JobPersistenceException: Couldn't retrieve
>>> trigger: 2 [See nested exception:
>>> java.lang.ArrayIndexOutOfBoundsException: 2]
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:834)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$2.execute(JobStoreSupport.java:806)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3729)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3725)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:802)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:625)
>>> ... 5 more
>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
>>> at org.postgresql.util.PGbytea.toBytes(PGbytea.java:74)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2253)
>>> at
>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
>>> at
>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>> at
>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>> at
>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>> at
>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>> ... 13 more
>>> --
>>>
>>> Need help with troubleshooting this because I can't seem to get any
>>> further.
>>>
>>> Thanks!
>>>
>>> On Tue, Dec 23, 2014 at 1:25 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>> Hi Guys,
>>>>
>>>> I am setting up a new satellite server with a external postgres db and
>>>> just
>>>> cloned my first CentOS base/extras channel. On the client side everything
>>>> seems to have gone smoothly except when I now run "yum repolist" I see 0
>>>> packages for these 2 channels even though there are packages in those
>>>> channels. One thing I have noticed is that I am seeing tons of the
>>>> following errors in my /var/log/rhn/rhn_taskomatic_daemon.log log file on
>>>> the spacewalk server:
>>>>
>>>> --
>>>> ..snip..
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | 2014-12-23 13:16:23,458
>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>> ERROR
>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>> misfires: Couldn't retrieve trigger: 2
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 |
>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>> nested
>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | Caused by:
>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | ... 4 more
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | 2014-12-23 13:20:23,458
>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>> ERROR
>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>> misfires: Couldn't retrieve trigger: 2
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 |
>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>> nested
>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | Caused by:
>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | ... 4 more
>>>> ..snip..
>>>> --
>>>>
>>>> Any help would be greatly appreciated.
>>>>
>>>> Thanks!
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list