[Spacewalk-list] Problem with new satellite server deployment

Wed Dec 24 00:55:54 UTC 2014

Okay, this seems to be a problem on the client side now.  If I disable
ssl the client sees all the packages in the channel.  Will keep
troubleshooting and update once I find out the cause.

On Tue, Dec 23, 2014 at 4:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
> Ahh I was able to manually run the scheduled task to generate repodata
> and now see that the repository metadata for my channels has been
> generated successfully in the rhn_taskomatic_daemon.log file.  However
> the client still shows 0 packages.  Now I am not sure what else to
> look for or what I might be missing.  Also, would like to know your
> thoughts about using a external postgres db or not.
>
> Thanks much.
>
> On Tue, Dec 23, 2014 at 4:06 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>> Thanks Paul.  I downloaded the jar file from postgresql's site, as per
>> your recommendation and updated the links under /usr/share/java and
>> restarted spacewalk-services.  I no longer see the errors in
>> rhn_taskomatic_daemon.log file but the client still shows 0 packages
>> in those 2 channels.  Is there something I need to do in order for the
>> repodata to be generated?  Am I missing a step?  Would it be
>> recommended to use a external postgres db or a embedded one?
>>
>> Thanks for all your help with this.  It is much appreciated!
>>
>> On Tue, Dec 23, 2014 at 3:30 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
>>> Yes absolutely I've hit this too.
>>>
>>> You need to upgrade the jdbc driver on the box running spacewalk to match
>>> the version of PostgreSQL. You can download it off the PostgreSQL web site.
>>> Its a jar file you need to download it to the correct directory then modify
>>> the symlinks to point to it.
>>>
>>> I've written instructions on how to do this in the past on this list if you
>>> can find it in google that's great. But if you can't let me know and I'll
>>> email the instructions when I get back in the office tomorrow morning.
>>>
>>>
>>>
>>> -- Sent from my HP Pre3
>>>
>>> ________________________________
>>> On Dec 23, 2014 6:02 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>
>>> Thanks for your reply Paul. Looks like my dab's are running version 9.3:
>>>
>>> /opt/postgres/pgserver9.3/bin/postgres
>>>
>>> You think this is related to the db side?
>>>
>>>
>>> On Tue, Dec 23, 2014 at 2:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>> Since I am not using this in production yet, I decided to upgrade
>>>> everything to the latest available on the spacewalk server running
>>>> CentOS. Did not touch the external postgres server running on RHEL.
>>>> However, still getting the same issue. This is the complete error
>>>> when in the rhn_taskomatic_daemon.log file when restarting spacewalk:
>>>>
>>>> --
>>>> STATUS | wrapper | 2014/12/23 14:09:31 | TERM trapped. Shutting down.
>>>> STATUS | wrapper | 2014/12/23 14:09:32 | <-- Wrapper Stopped
>>>> STATUS | wrapper | 2014/12/23 14:10:08 | --> Wrapper Started as Daemon
>>>> STATUS | wrapper | 2014/12/23 14:10:08 | Launching a JVM...
>>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Wrapper (Version 3.2.3)
>>>> http://wrapper.tanukisoftware.org
>>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Copyright 1999-2006 Tanuki
>>>> Software, Inc. All Rights Reserved.
>>>> INFO | jvm 1 | 2014/12/23 14:10:08 |
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>> com.mchange.v2.log.MLog <clinit>
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: MLog clients using
>>>> java 1.4+ standard logging.
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>> com.mchange.v2.c3p0.C3P0Registry banner
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing
>>>> c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5]
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing c3p0
>>>> pool... com.mchange.v2.c3p0.PoolBackedDataSource at 585a1c43 [
>>>> connectionPoolDataSource ->
>>>> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 1c136579 [
>>>> acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
>>>> -> 1000, autoCommitOnClose -> false, automaticTestTable -> null,
>>>> breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
>>>> connectionCustomizerClassName ->
>>>> com.redhat.rhn.common.db.RhnConnectionCustomizer,
>>>> connectionTesterClassName ->
>>>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>>>> debugUnreturnedConnectionStackTraces -> false, factoryClassLocation ->
>>>> null, forceIgnoreUnresolvedTransactions -> false, identityToken ->
>>>> k0jcdo96xmmxcv1leoshp|2d09bbf2, idleConnectionTestPeriod -> 300,
>>>> initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge
>>>> -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0,
>>>> maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection ->
>>>> 0, minPoolSize -> 5, nestedDataSource ->
>>>> com.mchange.v2.c3p0.DriverManagerDataSource at f7c08283 [ description ->
>>>> null, driverClass -> null, factoryClassLocation -> null, identityToken
>>>> -> k0jcdo96xmmxcv1leoshp|7451e191, jdbcUrl ->
>>>> jdbc:postgresql://rwcswalkpg01:5432/spaceschema, properties ->
>>>> {user=******, password=******, driver_proto=jdbc:postgresql} ],
>>>> preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle ->
>>>> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true,
>>>> unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies ->
>>>> false; userOverrides: {} ], dataSourceName -> null,
>>>> factoryClassLocation -> null, identityToken ->
>>>> k0jcdo96xmmxcv1leoshp|2fe19787, numHelperThreads -> 3 ]
>>>> FATAL | jvm 1 | 2014/12/23 14:10:16 | Failure occured during job recovery.
>>>> com.redhat.rhn.taskomatic.core.TaskomaticException: Failure occured
>>>> during job recovery.
>>>> at
>>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:158)
>>>> at
>>>> com.redhat.rhn.taskomatic.core.TaskomaticDaemon$1.run(TaskomaticDaemon.java:102)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.quartz.SchedulerConfigException: Failure occured during
>>>> job recovery. [See nested exception:
>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>> nested exception: java.lang.ArrayIndexOutOfBoundsException: 2]]
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:627)
>>>> at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:494)
>>>> at org.quartz.impl.StdScheduler.start(StdScheduler.java:143)
>>>> at
>>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:146)
>>>> ... 2 more
>>>> Caused by: org.quartz.JobPersistenceException: Couldn't retrieve
>>>> trigger: 2 [See nested exception:
>>>> java.lang.ArrayIndexOutOfBoundsException: 2]
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:834)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$2.execute(JobStoreSupport.java:806)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3729)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3725)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:802)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:625)
>>>> ... 5 more
>>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
>>>> at org.postgresql.util.PGbytea.toBytes(PGbytea.java:74)
>>>> at
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2253)
>>>> at
>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
>>>> at
>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>> at
>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>> ... 13 more
>>>> --
>>>>
>>>> Need help with troubleshooting this because I can't seem to get any
>>>> further.
>>>>
>>>> Thanks!
>>>>
>>>> On Tue, Dec 23, 2014 at 1:25 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>>> Hi Guys,
>>>>>
>>>>> I am setting up a new satellite server with a external postgres db and
>>>>> just
>>>>> cloned my first CentOS base/extras channel. On the client side everything
>>>>> seems to have gone smoothly except when I now run "yum repolist" I see 0
>>>>> packages for these 2 channels even though there are packages in those
>>>>> channels. One thing I have noticed is that I am seeing tons of the
>>>>> following errors in my /var/log/rhn/rhn_taskomatic_daemon.log log file on
>>>>> the spacewalk server:
>>>>>
>>>>> --
>>>>> ..snip..
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | 2014-12-23 13:16:23,458
>>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>>> ERROR
>>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>>> misfires: Couldn't retrieve trigger: 2
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 |
>>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>>> nested
>>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | Caused by:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | ... 4 more
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | 2014-12-23 13:20:23,458
>>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>>> ERROR
>>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>>> misfires: Couldn't retrieve trigger: 2
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 |
>>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>>> nested
>>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | Caused by:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | ... 4 more
>>>>> ..snip..
>>>>> --
>>>>>
>>>>> Any help would be greatly appreciated.
>>>>>
>>>>> Thanks!
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list