[Spacewalk-list] Problem with channel
Nicolas Michel
be.nicolas.michel at gmail.com
Thu Nov 20 10:07:39 UTC 2014
OK. I manage to find a fix, so I share it ;)
The problem was the cache of the channel. It was in status "in
progress" when checking it by the web interface. Then I checked in
/var/log/rhn/rhn_taskomatic_daemon.log and saw a posgresql out of
memory error (log extract is below). So I figured out a
misconfiguration from me of postgres and changed it there :
/var/lib/pgsql/data/postgresql.conf
After that, I restarted spacewalk-service and got another error about
an out of memory error, but this time in the JVM. I searched the web,
and call a collegue which is a java expert. we fix it by editing
/var/log/rhn/rhn_taskomatic_daemon.log and changed these values:
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=2048
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048
My collegue also pointed out what is according to him (and without him
knowing spacewalk at all before calling him) a misconfiguration in the
java configuration of the catalina server:
-XX:MaxNewSize=256
So I removed it and checking with jstat showed better memory behavior
according to him.
So, the final fix for my problem was to increase the memory of the
taskomatic JVM.
FIRST ERROR (postgres)
#######################
INFO | jvm 1 | 2014/11/20 09:27:32 | 2014-11-20 09:27:32,037
[Thread-51] INFO
com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
new repository metadata for channel 'rhel6-x86_64'(sha1)
22709 packages, 3014 errata
INFO | jvm 1 | 2014/11/20 09:27:52 | 2014-11-20 09:27:52,741
[Thread-51] ERROR com.redhat.rhn.common.db.datasource.CachedStatement
- Error while processing cached statement sql: SELECT p.id, pn.name as
n
ame, pevr.epoch as epoch,
INFO | jvm 1 | 2014/11/20 09:27:52 | pevr.version as
version, pevr.release as release,
INFO | jvm 1 | 2014/11/20 09:27:52 | p.summary,
p.description, pa.label as arch_label,
INFO | jvm 1 | 2014/11/20 09:27:52 | p.build_time,
p.path, p.package_size, p.payload_size, p.installed_size,
INFO | jvm 1 | 2014/11/20 09:27:52 | p.copyright,
p.vendor, p.build_host, p.header_start, p.header_end,
INFO | jvm 1 | 2014/11/20 09:27:52 | srpm.name as
source_rpm, pg.name as package_group_name,
INFO | jvm 1 | 2014/11/20 09:27:52 | cs.checksum,
cs.checksum_type as checksum_type,
INFO | jvm 1 | 2014/11/20 09:27:52 | prd.primary_xml as
primary_xml, prd.filelist as filelist_xml, prd.other as other_xml
INFO | jvm 1 | 2014/11/20 09:27:52 | FROM
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnChannelPackage cp,
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnPackage p
INFO | jvm 1 | 2014/11/20 09:27:52 | LEFT OUTER JOIN
rhnPackageGroup pg ON p.package_group = pg.id
INFO | jvm 1 | 2014/11/20 09:27:52 | LEFT OUTER JOIN
rhnSourceRpm srpm ON p.source_rpm_id = srpm.id
INFO | jvm 1 | 2014/11/20 09:27:52 | LEFT JOIN
rhnPackageRepodata prd ON prd.package_id = p.id,
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnPackageName pn,
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnPackageEvr pevr,
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnPackageArch pa,
INFO | jvm 1 | 2014/11/20 09:27:52 | rhnChecksumView cs
INFO | jvm 1 | 2014/11/20 09:27:52 | WHERE
INFO | jvm 1 | 2014/11/20 09:27:52 | cp.package_id = p.id
INFO | jvm 1 | 2014/11/20 09:27:52 | AND p.name_id = pn.id
INFO | jvm 1 | 2014/11/20 09:27:52 | AND p.evr_id = pevr.id
INFO | jvm 1 | 2014/11/20 09:27:52 | AND p.package_arch_id = pa.id
INFO | jvm 1 | 2014/11/20 09:27:52 | AND cp.channel_id = ?
INFO | jvm 1 | 2014/11/20 09:27:52 | AND p.checksum_id = cs.id
INFO | jvm 1 | 2014/11/20 09:27:52 | ORDER by 1
INFO | jvm 1 | 2014/11/20 09:27:52 |
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.translation.SqlExceptionTranslator.postgreSqlException(SqlExceptionTranslator.java:54)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.translation.SqlExceptionTranslator.sqlException(SqlExceptionTranslator.java:44)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:143)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:467)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
java.lang.Thread.run(Thread.java:744)
INFO | jvm 1 | 2014/11/20 09:27:52 | Caused by:
org.postgresql.util.PSQLException: Ran out of memory retrieving query
results.
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1777)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:360)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.execute(NewProxyPreparedStatement.java:362)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:140)
INFO | jvm 1 | 2014/11/20 09:27:52 | ... 11 more
INFO | jvm 1 | 2014/11/20 09:27:52 | Caused by:
java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349)
INFO | jvm 1 | 2014/11/20 09:27:52 | at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1774)
INFO | jvm 1 | 2014/11/20 09:27:52 | ... 17 more
INFO | jvm 1 | 2014/11/20 09:27:52 | 2014-11-20 09:27:52,742
[Thread-51] ERROR com.redhat.rhn.taskomatic.task.ChannelRepodata -
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.
INFO | jvm 1 | 2014/11/20 09:27:52 |
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.
SECOND ERROR - java out of memory
###################################
INFO | jvm 1 | 2014/11/20 10:19:05 | 2014-11-20 10:19:05,284
[Thread-53] INFO
com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
new repository metadata for channel 'rhel6-x86_64'(sha1) 22709
packages, 3014 errata
INFO | jvm 1 | 2014/11/20 10:19:37 | Exception in thread
"Thread-53" java.lang.OutOfMemoryError: GC overhead limit exceeded
INFO | jvm 1 | 2014/11/20 10:19:37 | at
java.lang.reflect.Method.copy(Method.java:151)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
java.lang.Class.copyMethods(Class.java:2852)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
java.lang.Class.getMethods(Class.java:1467)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.util.MethodUtil.callMethod(MethodUtil.java:118)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:776)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.processResultSet(CachedStatement.java:626)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:474)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO | jvm 1 | 2014/11/20 10:19:37 | at
java.lang.Thread.run(Thread.java:744)
2014-11-20 9:25 GMT+01:00 Nicolas Michel <be.nicolas.michel at gmail.com>:
> Hello guys,
>
> I have a problem with a channel: the packages are not available from
> the server clients.
>
> [root at my_hostname ~]# yum repolist
> Loaded plugins: product-id, rhnplugin, security, subscription-manager
> This system is not registered to Red Hat Subscription Management. You
> can use subscription-manager to register.
> This system is receiving updates from RHN Classic or Red Hat Satellite.
> repo id repo name
> status
> epel-rhel6-x86_64 EPEL-RHEL6-x86_64
> 13,045
> rhel6-x86_64 RHEL6-x86_64
> 0
> spacewalk-client-rhel6-x86_64 Spacewalk-Client-RHEL6-x86_64 44
> repolist: 13,089
>
> As you can see, the channel rhel6-x86_64 has no packages associated to
> itself, from the point of view of the client.
>
> From the spacewalk web interface, I do see that there are 22709
> packages associated to rhel6-x86_64.
>
> Channel Name Packages
> RHEL6-x86_64 22709
> EPEL-RHEL6-x86_64 13045
> HP-SDR-rhel-x86_64 375
> Spacewalk-Client-RHEL6-x86_64 44
>
> The fact is : my channel rhel6-x86_64 is the parent of the others. Is
> it a problem?
>
> --
> Nicolas MICHEL
--
Nicolas MICHEL
More information about the Spacewalk-list
mailing list