[Spacewalk-list] Problem with new satellite server deployment

Wed Dec 24 01:42:16 UTC 2014

This was a problem with my ssl certificate and the non fqdn name I
used for the ssl-common-name.  After re-generating this cert and
updating it on the client things seem to be good now.

Still thinking if I should use the external postgres db or use the embedded one.

Thanks!

On Tue, Dec 23, 2014 at 4:55 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
> Okay, this seems to be a problem on the client side now.  If I disable
> ssl the client sees all the packages in the channel.  Will keep
> troubleshooting and update once I find out the cause.
>
>
> On Tue, Dec 23, 2014 at 4:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>> Ahh I was able to manually run the scheduled task to generate repodata
>> and now see that the repository metadata for my channels has been
>> generated successfully in the rhn_taskomatic_daemon.log file.  However
>> the client still shows 0 packages.  Now I am not sure what else to
>> look for or what I might be missing.  Also, would like to know your
>> thoughts about using a external postgres db or not.
>>
>> Thanks much.
>>
>> On Tue, Dec 23, 2014 at 4:06 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>> Thanks Paul.  I downloaded the jar file from postgresql's site, as per
>>> your recommendation and updated the links under /usr/share/java and
>>> restarted spacewalk-services.  I no longer see the errors in
>>> rhn_taskomatic_daemon.log file but the client still shows 0 packages
>>> in those 2 channels.  Is there something I need to do in order for the
>>> repodata to be generated?  Am I missing a step?  Would it be
>>> recommended to use a external postgres db or a embedded one?
>>>
>>> Thanks for all your help with this.  It is much appreciated!
>>>
>>> On Tue, Dec 23, 2014 at 3:30 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
>>>> Yes absolutely I've hit this too.
>>>>
>>>> You need to upgrade the jdbc driver on the box running spacewalk to match
>>>> the version of PostgreSQL. You can download it off the PostgreSQL web site.
>>>> Its a jar file you need to download it to the correct directory then modify
>>>> the symlinks to point to it.
>>>>
>>>> I've written instructions on how to do this in the past on this list if you
>>>> can find it in google that's great. But if you can't let me know and I'll
>>>> email the instructions when I get back in the office tomorrow morning.
>>>>
>>>>
>>>>
>>>> -- Sent from my HP Pre3
>>>>
>>>> ________________________________
>>>> On Dec 23, 2014 6:02 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>>
>>>> Thanks for your reply Paul. Looks like my dab's are running version 9.3:
>>>>
>>>> /opt/postgres/pgserver9.3/bin/postgres
>>>>
>>>> You think this is related to the db side?
>>>>
>>>>
>>>> On Tue, Dec 23, 2014 at 2:14 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>>> Since I am not using this in production yet, I decided to upgrade
>>>>> everything to the latest available on the spacewalk server running
>>>>> CentOS. Did not touch the external postgres server running on RHEL.
>>>>> However, still getting the same issue. This is the complete error
>>>>> when in the rhn_taskomatic_daemon.log file when restarting spacewalk:
>>>>>
>>>>> --
>>>>> STATUS | wrapper | 2014/12/23 14:09:31 | TERM trapped. Shutting down.
>>>>> STATUS | wrapper | 2014/12/23 14:09:32 | <-- Wrapper Stopped
>>>>> STATUS | wrapper | 2014/12/23 14:10:08 | --> Wrapper Started as Daemon
>>>>> STATUS | wrapper | 2014/12/23 14:10:08 | Launching a JVM...
>>>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Wrapper (Version 3.2.3)
>>>>> http://wrapper.tanukisoftware.org
>>>>> INFO | jvm 1 | 2014/12/23 14:10:08 | Copyright 1999-2006 Tanuki
>>>>> Software, Inc. All Rights Reserved.
>>>>> INFO | jvm 1 | 2014/12/23 14:10:08 |
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>>> com.mchange.v2.log.MLog <clinit>
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: MLog clients using
>>>>> java 1.4+ standard logging.
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>>> com.mchange.v2.c3p0.C3P0Registry banner
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing
>>>>> c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5]
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | Dec 23, 2014 2:10:12 PM
>>>>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
>>>>> INFO | jvm 1 | 2014/12/23 14:10:12 | INFO: Initializing c3p0
>>>>> pool... com.mchange.v2.c3p0.PoolBackedDataSource at 585a1c43 [
>>>>> connectionPoolDataSource ->
>>>>> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource at 1c136579 [
>>>>> acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
>>>>> -> 1000, autoCommitOnClose -> false, automaticTestTable -> null,
>>>>> breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
>>>>> connectionCustomizerClassName ->
>>>>> com.redhat.rhn.common.db.RhnConnectionCustomizer,
>>>>> connectionTesterClassName ->
>>>>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>>>>> debugUnreturnedConnectionStackTraces -> false, factoryClassLocation ->
>>>>> null, forceIgnoreUnresolvedTransactions -> false, identityToken ->
>>>>> k0jcdo96xmmxcv1leoshp|2d09bbf2, idleConnectionTestPeriod -> 300,
>>>>> initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge
>>>>> -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0,
>>>>> maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection ->
>>>>> 0, minPoolSize -> 5, nestedDataSource ->
>>>>> com.mchange.v2.c3p0.DriverManagerDataSource at f7c08283 [ description ->
>>>>> null, driverClass -> null, factoryClassLocation -> null, identityToken
>>>>> -> k0jcdo96xmmxcv1leoshp|7451e191, jdbcUrl ->
>>>>> jdbc:postgresql://rwcswalkpg01:5432/spaceschema, properties ->
>>>>> {user=******, password=******, driver_proto=jdbc:postgresql} ],
>>>>> preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle ->
>>>>> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true,
>>>>> unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies ->
>>>>> false; userOverrides: {} ], dataSourceName -> null,
>>>>> factoryClassLocation -> null, identityToken ->
>>>>> k0jcdo96xmmxcv1leoshp|2fe19787, numHelperThreads -> 3 ]
>>>>> FATAL | jvm 1 | 2014/12/23 14:10:16 | Failure occured during job recovery.
>>>>> com.redhat.rhn.taskomatic.core.TaskomaticException: Failure occured
>>>>> during job recovery.
>>>>> at
>>>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:158)
>>>>> at
>>>>> com.redhat.rhn.taskomatic.core.TaskomaticDaemon$1.run(TaskomaticDaemon.java:102)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: org.quartz.SchedulerConfigException: Failure occured during
>>>>> job recovery. [See nested exception:
>>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>>> nested exception: java.lang.ArrayIndexOutOfBoundsException: 2]]
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:627)
>>>>> at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:494)
>>>>> at org.quartz.impl.StdScheduler.start(StdScheduler.java:143)
>>>>> at
>>>>> com.redhat.rhn.taskomatic.core.SchedulerKernel.startup(SchedulerKernel.java:146)
>>>>> ... 2 more
>>>>> Caused by: org.quartz.JobPersistenceException: Couldn't retrieve
>>>>> trigger: 2 [See nested exception:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 2]
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:834)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$2.execute(JobStoreSupport.java:806)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3729)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3725)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:802)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:625)
>>>>> ... 5 more
>>>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
>>>>> at org.postgresql.util.PGbytea.toBytes(PGbytea.java:74)
>>>>> at
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2253)
>>>>> at
>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
>>>>> at
>>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>>> at
>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>>> ... 13 more
>>>>> --
>>>>>
>>>>> Need help with troubleshooting this because I can't seem to get any
>>>>> further.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Tue, Dec 23, 2014 at 1:25 PM, Jagga Soorma <jagga13 at gmail.com> wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> I am setting up a new satellite server with a external postgres db and
>>>>>> just
>>>>>> cloned my first CentOS base/extras channel. On the client side everything
>>>>>> seems to have gone smoothly except when I now run "yum repolist" I see 0
>>>>>> packages for these 2 channels even though there are packages in those
>>>>>> channels. One thing I have noticed is that I am seeing tons of the
>>>>>> following errors in my /var/log/rhn/rhn_taskomatic_daemon.log log file on
>>>>>> the spacewalk server:
>>>>>>
>>>>>> --
>>>>>> ..snip..
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | 2014-12-23 13:16:23,458
>>>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>>>> ERROR
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>>>> misfires: Couldn't retrieve trigger: 2
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 |
>>>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>>>> nested
>>>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | Caused by:
>>>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>>>> INFO | jvm 1 | 2014/12/23 13:16:23 | ... 4 more
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | 2014-12-23 13:20:23,458
>>>>>> [QuartzScheduler_DefaultQuartzScheduler-NON_CLUSTERED_MisfireHandler]
>>>>>> ERROR
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreTX - MisfireHandler: Error handling
>>>>>> misfires: Couldn't retrieve trigger: 2
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 |
>>>>>> org.quartz.JobPersistenceException: Couldn't retrieve trigger: 2 [See
>>>>>> nested
>>>>>> exception: java.lang.ArrayIndexOutOfBoundsException: 2]
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1571)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverMisfiredJobs(JobStoreSupport.java:950)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3125)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3896)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3916)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | Caused by:
>>>>>> java.lang.ArrayIndexOutOfBoundsException: 2
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>> org.postgresql.util.PGbytea.toBytes(PGbytea.java:76)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2451)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.apache.commons.dbcp.DelegatingResultSet.getBytes(DelegatingResultSet.java:253)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.PostgreSQLDelegate.getObjectFromBlob(PostgreSQLDelegate.java:92)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectTrigger(StdJDBCDelegate.java:2132)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | at
>>>>>>
>>>>>> org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1553)
>>>>>> INFO | jvm 1 | 2014/12/23 13:20:23 | ... 4 more
>>>>>> ..snip..
>>>>>> --
>>>>>>
>>>>>> Any help would be greatly appreciated.
>>>>>>
>>>>>> Thanks!
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list