[Spacewalk-list] Taskomatic runs indefinitely without ever generating repodata
Gerald Vogt
vogt at spamcop.net
Fri Jul 6 07:20:56 UTC 2018
On 05.07.18 21:30, Matt Moldvan wrote:
> Is there anything interesting
> in /var/log/rhn/tasko/sat/channel-repodata-bunch? Do you have any hung
There is currently only a single file with this content:
spacewalk:channel-repodata-bunch(996)# ls -l
total 4
-rw-r--r--. 1 root root 130 Jul 2 08:13 channel-repodata_15408487_out
spacewalk:channel-repodata-bunch(997)# cat channel-repodata_15408487_out
2018-07-02 08:13:10,793 [DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
> reposync processes? Any lingering Postgres locks that might be an issue?
No reposync processes. All progres processes say "idle", so I guess there are no locks. Or how do I check for lingering locks?
> It's odd that the run would only take 1 second, unless something is
> wrong with the database or it's data...
>
> What do you see from a spacewalk-sql command like below?
I see all the channels:
label | name | modified | last_synced
-----------------------------------+------------------------------------------------+-------------------------------+----------------------------
icinga-epel7-x86_64 | ICINGA stable release for epel-7 (x86_64) | 2016-02-15 10:07:59.822942+01 | 2018-07-06 02:30:55.482+02
epel7-centos7-x86_64 | EPEL 7 for CentOS 7 (x86_64) | 2014-07-21 08:16:26.367135+02 | 2018-07-06 04:01:52.148+02
centos6-x86_64-extras | CentOS 6 Extras (x86_64) | 2012-08-23 06:46:05.145629+02 | 2018-06-21 10:25:26.104+02
grafana-epe7-x86_64 | Grafana stable release for epel-7 (x86_64) | 2016-05-06 08:29:49.308149+02 | 2018-06-21 04:58:15.022+02
spacewalk26-client-centos6-x86_64 | Spacewalk Client 2.6 for CentOS 6 (x86_64) | 2017-04-25 13:44:49.266738+02 | 2018-06-21 10:41:07.369+02
globus-el6-x86_64 | Globus Toolkit 6 (el6) | 2016-05-13 15:23:31.807011+02 | 2018-07-06 03:34:49.95+02
internet2 | perfSONAR RPM Repository | 2017-06-27 06:56:33.675378+02 | 2018-06-22 10:24:41.702+02
postgresql94-centos6-x86_64 | PostgreSQL 9.4 for CentOS 6 (x86_64) | 2015-01-28 14:09:41.856451+01 | 2018-06-21 10:42:01.413+02
spacewalk26-server-centos6-x86_64 | Spacewalk Server 2.6 for CentOS 6 (x86_64) | 2017-04-25 13:39:38.250769+02 | 2018-06-21 10:36:17.46+02
centos7-x86_64-fasttrack | CentOS 7 FastTrack (x86_64) | 2014-07-21 08:16:26.017642+02 | 2018-06-21 10:26:29.571+02
spacewalk26-client-centos7-x86_64 | Spacewalk Client 2.6 for CentOS 7 (x86_64) | 2017-04-25 13:46:00.107344+02 | 2018-06-22 10:22:28.484+02
centos7-x86_64-centosplus | CentOS 7 Plus (x86_64) | 2014-07-21 08:16:25.467309+02 | 2018-06-21 10:25:19.884+02
centos6-x86_64-centosplus | CentOS 6 Plus (x86_64) | 2012-08-23 07:18:00.349338+02 | 2018-06-21 10:36:04.08+02
docker-ce-centos7-x86_64 | Docker CE Stable for CentOS 7 (x86_64) | 2017-09-28 12:52:45.858354+02 | 2018-07-06 04:30:05.442+02
postgresql10-centos7-x86_64 | PostgreSQL 10 for CentOS 7 (x86_64) | 2018-02-12 14:48:14.617235+01 | 2018-02-12 15:06:16.464+01
bareos162-centos7-x86_64 | Bareos 16.2 for CentOS 7 (x86_64) | 2017-09-26 14:37:16.533773+02 | 2018-06-21 04:59:21.954+02
docker-ce-edge-centos7-x86_64 | Docker CE Edge for CentOS 7 (x86_64) | 2017-12-29 09:58:14.581069+01 | 2018-06-21 04:59:39.796+02
beegfs6-centos7-x86_64 | BeeGFS 6 for CentOS 7 (x86_64) | 2018-03-19 14:08:08.389588+01 | 2018-06-21 04:59:43.132+02
icinga-epel6-x86_64 | ICINGA stable release for epel-6 (x86_64) | 2018-01-15 15:41:31.138875+01 | 2018-07-06 02:30:28.142+02
openstack-pike-centos7 | OpenStack Pike for CentOS 7 | 2017-10-05 09:10:22.575224+02 | 2018-06-21 05:36:35.43+02
globus-el7-x86_64 | Globus Toolkit 6 (el7) | 2017-09-28 13:00:07.32028+02 | 2018-07-06 03:31:22.806+02
postgresql10-centos6-x86_64 | PostgreSQL 10 for CentOS 6 (x86_64) | 2018-02-12 14:48:55.970013+01 | 2018-07-06 04:02:04.03+02
ceph-jewel-centos7 | CentOS 7 Ceph Jewel (x86_64) | 2018-02-12 12:15:28.8976+01 | 2018-07-06 05:30:07.085+02
spacewalk28-server-centos6-x86_64 | Spacewalk Server 2.8 for CentOS 6 (x86_64) | 2018-06-22 18:05:55.190988+02 | 2018-07-06 06:18:15.016+02
spacewalk28-client-centos7-x86_64 | Spacewalk Client 2.8 for CentOS 7 (x86_64) | 2018-06-22 18:05:55.575963+02 | 2018-07-06 06:18:22.41+02
puppet5-el7-x86_64 | Puppet 5 for EL 7 (x86_64) | 2018-03-28 14:20:52.254978+02 | 2018-06-21 06:01:31.357+02
centos7-qemu-ev | CentOS 7 QEMU EV (x86_64) | 2018-02-12 12:15:06.116673+01 | 2018-07-06 05:30:12.078+02
bareos172-centos7-x86_64 | Bareos 17.2 for CentOS 7 (x86_64) | 2018-05-08 14:18:56.708206+02 | 2018-06-22 10:24:48.431+02
openstack-queens-centos7 | OpenStack Queens for CentOS 7 | 2018-03-28 13:08:27.607498+02 | 2018-07-06 05:30:44.123+02
elrepo-centos7 | ELRepo for CentOS 7 | 2017-09-18 12:03:42.302442+02 | 2018-06-21 05:01:30.303+02
spacewalk28-client-centos6-x86_64 | Spacewalk Client 2.8 for CentOS 6 (x86_64) | 2018-06-22 18:05:53.475193+02 | 2018-07-06 06:18:07.158+02
centos7-x86_64-extras | CentOS 7 Extras (x86_64) | 2014-07-21 08:16:25.841121+02 | 2018-06-21 10:26:27.879+02
internet2-web100_kernel | perfSONAR Web100 Kernel RPM Repository | 2017-06-27 06:57:03.825602+02 | 2018-06-21 10:24:50.96+02
centos6-x86_64-updates | CentOS 6 Updates (x86_64) | 2012-08-23 06:46:05.264195+02 | 2018-06-21 10:34:24.866+02
centos7-x86_64-updates | CentOS 7 Updates (x86_64) | 2014-07-21 08:16:26.196397+02 | 2018-07-02 09:48:02.273+02
centos6-x86_64-fasttrack | CentOS 6 FastTrack (x86_64) | 2012-08-23 06:46:05.205228+02 | 2018-06-22 10:24:43.51+02
postgresql92-centos6-x86_64 | PostgreSQL 9.2 for CentOS 6 (x86_64) | 2012-09-12 08:15:27.194188+02 | 2018-07-06 03:47:12.311+02
epel6-centos6-x86_64 | EPEL 6 for CentOS 6 (x86_64) | 2012-08-23 06:46:30.597753+02 | 2018-07-06 03:55:48.834+02
jpackage5.0-generic | JPackage 5.0 for generic | 2014-07-02 10:32:24.985979+02 | 2018-07-06 03:46:46.084+02
hp-spp-rhel-7 | HP Software Delivery Repository for SPP RHEL 7 | 2015-04-16 14:18:33.041249+02 | 2018-07-06 05:31:01.633+02
owncloud-centos7-noarch | ownCloud for CentOS 7 | 2015-01-28 13:53:41.415573+01 | 2018-06-21 05:36:40.901+02
centos7-x86_64-scl | CentOS 7 SCL (x86_64) | 2016-04-15 11:26:29.042925+02 | 2018-06-21 05:14:49.359+02
postgresql96-centos6-x86_64 | PostgreSQL 9.6 for CentOS 6 (x86_64) | 2017-02-09 15:31:54.632728+01 | 2018-06-21 05:00:39.353+02
postgresql96-centos7-x86_64 | PostgreSQL 9.6 for CentOS 7 (x86_64) | 2017-02-09 15:35:13.136001+01 | 2018-07-06 04:30:19.645+02
centos6-x86_64 | CentOS 6 (x86_64) | 2012-08-23 06:46:04.610089+02 | 2018-07-05 22:03:30.089+02
centos7-x86_64 | CentOS 7 (x86_64) | 2014-07-21 08:16:24.172395+02 | 2018-07-05 22:08:45.242+02
-Gerald
>
> echo 'select label,name,modified,last_synced from rhnchannel' | sudo
> spacewalk-sql -i
>
> label | name | modified|last_synced
>
> ----------------------------------+----------------------------------+-------------------------------+----------------------------
>
> ovirt-x86_64-stable-6-nonprod| ovirt-x86_64-stable-6-nonprod| 2015-09-14
> 13:46:44.147134-05 |
>
> extras7-x86_64-nonprod | extras7-x86_64-nonprod | 2017-11-06
> 10:26:30.011283-06 |
>
> centos7-x86_64-all | centos7-x86_64-all | 2015-11-11 08:50:58.831234-06
> | 2018-07-05 11:01:08.857-05
>
> perl-5.16.x-all| perl-5.16.x-all| 2015-09-11 13:25:15.002198-05 |
> 2015-09-11 13:29:21.361-05
>
> ovirt-x86_64-stable-6| ovirt-x86_64-stable-6| 2015-09-14 13:30:55.172-05|
>
> ovirt-x86_64-stable-6-prod | ovirt-x86_64-stable-6-prod | 2015-09-14
> 13:48:06.637063-05 |
>
> other6-x86_64-all| other6-x86_64-all| 2015-07-28 09:20:38.156104-05 |
>
> epel5-x86_64-all | epel5-x86_64-all | 2016-10-04 18:20:44.846312-05 |
> 2017-04-17 12:57:36.859-05
>
> passenger6-x86_64-prod | passenger6-x86_64-prod | 2016-04-22
> 14:35:45.395518-05 |
>
> perl-5.16.x-nonprod| perl-5.16.x-nonprod| 2015-09-11 13:27:32.261063-05 |
>
> perl-5.16.x-prod | perl-5.16.x-prod | 2015-09-11 13:26:40.584715-05 |
> 2015-09-11 13:29:38.537-05
>
> other6-x86_64-nonprod| other6-x86_64-nonprod| 2015-07-23
> 15:00:03.733479-05 |
>
> other6-x86_64-prod | other6-x86_64-prod | 2015-07-21 15:10:48.719528-05 |
>
> epel5-x86_64-prod| epel5-x86_64-prod| 2016-10-04 18:25:38.655383-05 |
>
> passenger6-x86_64-all| passenger6-x86_64-all| 2016-04-20
> 11:37:19.002493-05 | 2016-04-20 11:58:42.312-05
>
> docker7-x86_64-prod| docker7-x86_64-prod| 2017-08-03 11:42:08.474496-05 |
>
> centos5-x86_64-nonprod | centos5-x86_64-nonprod | 2015-06-22
> 16:16:17.372799-05 |
>
> other7-x86_64-nonprod| other7-x86_64-nonprod| 2016-07-14
> 13:03:10.320136-05 |
>
> mongo3.2-centos6-x86_64-all| mongo3.2-centos6-x86_64-all| 2016-08-22
> 12:21:40.722182-05 | 2018-07-01 12:27:03.019-05
>
> centos5-x86_64-prod| centos5-x86_64-prod| 2015-06-22 16:20:41.474486-05 |
>
> passenger6-x86_64-nonprod| passenger6-x86_64-nonprod| 2016-04-20
> 12:29:24.677227-05 |
>
> other7-x86_64-prod | other7-x86_64-prod | 2016-07-14 13:03:47.284295-05 |
>
> cloudera5.7-x86_64-nonprod | cloudera5.7-x86_64-nonprod | 2016-05-09
> 12:10:16.496626-05 | 2016-06-20 13:11:20.62-05
>
> epel5-x86_64-nonprod | epel5-x86_64-nonprod | 2016-10-04
> 18:25:09.844486-05 |
>
> epel6-x86_64-prod| epel6-x86_64-prod| 2016-03-18 11:52:45.9199-05 |
> 2016-08-23 05:07:37.967-05
>
> spacewalk6-client-all| spacewalk6-client-all| 2017-05-02
> 20:53:38.867018-05 | 2018-07-01 22:02:11.386-05
>
> docker7-x86_64-nonprod | docker7-x86_64-nonprod | 2017-04-07
> 15:13:44.158973-05 |
>
> mongo3.2-centos6-x86_64-nonprod| mongo3.2-centos6-x86_64-nonprod|
> 2016-08-22 12:34:18.095059-05 |
>
> mongo3.2-centos6-x86_64-prod | mongo3.2-centos6-x86_64-prod | 2016-08-22
> 12:42:19.161165-05 |
>
> local6-x86_64-all| local6-x86_64-all| 2015-09-30 08:55:37.657412-05 |
> 2016-04-19 07:00:23.632-05
>
> centos5-x86_64-all | centos5-x86_64-all | 2015-06-22 15:20:22.085465-05
> | 2017-04-17 13:09:39.635-05
>
> spacewalk5-client-nonprod| spacewalk5-client-nonprod| 2017-05-02
> 20:53:20.430795-05 |
>
> spacewalk5-client-prod | spacewalk5-client-prod | 2017-05-02
> 20:53:28.980968-05 |
>
> spacewalk5-client-all| spacewalk5-client-all| 2017-05-02
> 20:53:08.276664-05 | 2018-07-05 10:10:11.665-05
>
> spacewalk7-client-prod | spacewalk7-client-prod | 2017-05-02
> 20:54:32.321635-05 | 2018-07-05 11:01:14.499-05
>
> epel6-x86_64-nonprod | epel6-x86_64-nonprod | 2016-03-18
> 11:52:14.915108-05 | 2018-07-05 10:10:08.774-05
>
> centos7-x86_64-prod| centos7-x86_64-prod| 2015-11-11 09:02:06.69758-06|
>
> puppetlabs6-x86_64-prod| puppetlabs6-x86_64-prod| 2016-04-22
> 13:46:22.233841-05 | 2018-07-01 13:30:47.635-05
>
> puppetlabs5-x86_64-nonprod | puppetlabs5-x86_64-nonprod | 2018-03-26
> 15:21:59.007749-05 | 2018-07-01 13:00:03.401-05
>
> puppetlabs5-x86_64-prod| puppetlabs5-x86_64-prod| 2018-03-26
> 15:24:23.86552-05| 2018-07-01 13:30:39.025-05
>
> puppetlabs5-x86_64-all | puppetlabs5-x86_64-all | 2018-03-26
> 15:19:04.647981-05 | 2018-07-01 13:31:25.065-05
>
> other5-x86_64-all| other5-x86_64-all| 2015-08-10 14:16:01.092867-05 |
>
> other5-x86_64-nonprod| other5-x86_64-nonprod| 2015-08-10
> 14:18:05.114541-05 |
>
> other5-x86_64-prod | other5-x86_64-prod | 2015-08-10 14:19:03.728982-05 |
>
> centos6-x86_64-nonprod | centos6-x86_64-nonprod | 2015-06-22
> 16:24:07.137207-05 |
>
> centos6-x86_64-prod| centos6-x86_64-prod| 2015-06-22 16:28:51.324002-05 |
>
> extras7-x86_64-all | extras7-x86_64-all | 2017-08-16 09:13:26.8122-05 |
> 2018-07-05 10:05:10.626-05
>
> centos6-x86_64-gitlab-ce-nonprod | centos6-x86_64-gitlab-ce-nonprod |
> 2017-04-17 11:43:36.609036-05 | 2018-07-05 10:04:57.277-05
>
> spacewalk7-server-all| spacewalk7-server-all| 2017-03-28
> 15:22:31.851414-05 | 2018-07-05 11:11:31.564-05
>
> local5-x86_64-all| local5-x86_64-all| 2016-02-24 12:19:36.791459-06 |
>
> local5-x86_64-nonprod| local5-x86_64-nonprod| 2016-02-24
> 12:20:19.404008-06 |
>
> local5-x86_64-prod | local5-x86_64-prod | 2016-02-24 12:20:45.098532-06 |
>
> local6-x86_64-nonprod| local6-x86_64-nonprod| 2016-08-22 20:49:56.7376-05 |
>
> local7-x86_64-all| local7-x86_64-all| 2016-07-14 13:00:32.511851-05 |
>
> local7-x86_64-nonprod| local7-x86_64-nonprod| 2016-07-14
> 13:02:06.932169-05 |
>
> local7-x86_64-prod | local7-x86_64-prod | 2016-07-14 13:02:38.496912-05 |
>
> puppetlabs6-x86_64-all | puppetlabs6-x86_64-all | 2016-04-20
> 08:27:56.026914-05 | 2018-07-01 13:30:36.771-05
>
> spacewalk7-client-nonprod| spacewalk7-client-nonprod| 2017-05-02
> 20:54:22.659512-05 | 2018-07-05 11:10:25.009-05
>
> docker7-x86_64-all | docker7-x86_64-all | 2017-03-22 12:50:15.332561-05
> | 2018-07-05 13:00:02.988-05
>
> spacewalk7-client-all| spacewalk7-client-all| 2017-05-02
> 20:54:13.5076-05 | 2018-07-05 10:04:59.748-05
>
> local6-x86_64-prod | local6-x86_64-prod | 2015-09-30 08:59:12.679727-05 |
>
> centos6-x86_64-gitlab-ee-nonprod | centos6-x86_64-gitlab-ee-nonprod |
> 2016-04-14 11:39:01.432444-05 | 2018-07-05 11:12:20.525-05
>
> mysqltools6-x86_64-all | mysqltools6-x86_64-all | 2016-03-17
> 12:41:37.44854-05| 2018-07-05 12:00:02.319-05
>
> mysqltools6-x86_64-nonprod | mysqltools6-x86_64-nonprod | 2016-03-17
> 12:58:35.036373-05 |
>
> mysqltools6-x86_64-prod| mysqltools6-x86_64-prod| 2016-03-17
> 12:59:10.969162-05 |
>
> spacewalk7-server-nonprod| spacewalk7-server-nonprod| 2017-03-28
> 15:23:02.210349-05 | 2018-07-05 11:12:47.471-05
>
> spacewalk7-server-prod | spacewalk7-server-prod | 2017-03-28
> 15:23:29.309042-05 | 2017-05-02 20:56:45.247-05
>
> epel7-x86_64-prod| epel7-x86_64-prod| 2016-03-22 09:48:38.060213-05 |
> 2018-07-05 09:57:25.861-05
>
> puppetlabs6-x86_64-nonprod | puppetlabs6-x86_64-nonprod | 2016-04-20
> 12:28:55.337125-05 | 2018-07-01 13:30:43.362-05
>
> newrelic-noarch-nover| newrelic-noarch-nover| 2016-10-13
> 13:54:38.621333-05 | 2016-10-13 14:09:41.778-05
>
> other7-x86_64-all| other7-x86_64-all| 2016-07-14 13:01:25.848215-05 |
> 2018-07-05 14:00:03.714-05
>
> spacewalk6-client-nonprod| spacewalk6-client-nonprod| 2017-05-02
> 20:53:50.507298-05 |
>
> spacewalk6-client-prod | spacewalk6-client-prod | 2017-05-02
> 20:54:00.685324-05 |
>
> spacewalk6-server-all| spacewalk6-server-all| 2018-06-22
> 23:11:30.637054-05 | 2018-07-05 11:01:11.543-05
>
> puppetlabs7-x86_64-prod| puppetlabs7-x86_64-prod| 2016-07-14
> 13:29:04.67033-05| 2018-07-01 13:31:29.425-05
>
> spacewalk6-server-nonprod| spacewalk6-server-nonprod| 2018-06-22
> 23:17:20.660409-05 |
>
> spacewalk6-server-prod | spacewalk6-server-prod | 2018-06-22
> 23:18:02.738869-05 |
>
> puppetlabs7-x86_64-nonprod | puppetlabs7-x86_64-nonprod | 2016-07-14
> 13:28:34.475051-05 | 2018-07-01 13:16:25.948-05
>
> epel6-x86_64-all | epel6-x86_64-all | 2016-03-18 11:50:17.587171-05 |
> 2018-07-05 11:07:42.644-05
>
> centos6-x86_64-gitlab-ee | centos6-x86_64-gitlab-ee | 2015-12-24
> 13:21:10.493684-06 | 2018-07-05 11:08:30.039-05
>
> puppetlabs7-x86_64-all | puppetlabs7-x86_64-all | 2016-07-14
> 12:54:59.388232-05 | 2018-07-01 13:32:02.745-05
>
> epel7-x86_64-nonprod | epel7-x86_64-nonprod | 2016-03-22
> 09:47:34.668867-05 | 2017-04-21 11:08:24.573-05
>
> centos6-x86_64-all | centos6-x86_64-all | 2015-06-22 15:19:13.053429-05
> | 2018-07-02 01:12:57.768-05
>
> epel7-x86_64-all | epel7-x86_64-all | 2016-03-22 09:44:48.748142-05 |
> 2018-07-05 09:11:28.553-05
>
> centos7-x86_64-nonprod | centos7-x86_64-nonprod | 2015-10-21
> 22:02:28.107902-05 |
>
> (85 rows)
>
>
> On Thu, Jul 5, 2018 at 11:48 AM Gerald Vogt <vogt at spamcop.net
> <mailto:vogt at spamcop.net>> wrote:
>
> On 05.07.18 16:05, Matt Moldvan wrote:
> > How is the server utilization with respect to disk I/O (something
> like
> > iotop or htop might help here)? Maybe there is something else
> blocking
>
> My server is basically idle. 99% idle, little disk i/o. It doesn't
> do anything really.
>
> > and the server doesn't have enough resources to complete. Have you
> > tried running an strace against the running process?
>
> If it doesn't have enough resources shouldn't there be an exception?
>
> For me, it looks more like something doesn't make it into the
> database and thus into the persistent state. For instance, I now
> have the repodata task at "RUNNING" for three days:
>
> Channel Repodata: 2018-07-02 08:13:10 CEST RUNNING
>
> The log file shows this regarding repodata:
>
> > # fgrep -i repodata rhn_taskomatic_daemon.log
> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,584
> [Thread-12] INFO com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
> single-channel-repodata-bunch-0 scheduled succesfully.
> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,636
> [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,651
> [DefaultQuartzScheduler_Worker-8] DEBUG
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: task channel-repodata started
> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,793
> [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,102
> [DefaultQuartzScheduler_Worker-8] DEBUG
> com.redhat.rhn.taskomatic.TaskoJob - channel-repodata
> (single-channel-repodata-bunch-0) ... running
> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,103
> [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: bunch channel-repodata-bunch FINISHED
>
> So according to the logs the repodata bunch has finished. According
> to the web interface it has not. Nothing has been updated in
> /var/cache/rhn/repodata/ either. In addition, those four channels
> which were still updated haven't been updated either now.
>
> Thanks,
>
> Gerald
>
>
>
> >
> > I also had an (well, many) issue(s) with our Spacewalk server before
> > disabling snapshots in /etc/rhn/rhn.conf. I also increased the
> number
> > of workers and max repodata work items:
> >
> > # system snapshots enabled
> > enable_snapshots = 0
> > ...
> > taskomatic.maxmemory=6144
> > taskomatic.errata_cache_max_work_items = 500
> > taskomatic.channel_repodata_max_work_items = 50
> > taskomatic.channel_repodata_workers = 5
> >
> >
> >
> > On Thu, Jul 5, 2018 at 4:38 AM Florence Savary
> > <florence.savary.fs at gmail.com
> <mailto:florence.savary.fs at gmail.com>
> <mailto:florence.savary.fs at gmail.com
> <mailto:florence.savary.fs at gmail.com>>> wrote:
> >
> > Hello,
> >
> > Thanks for sharing your configuration files. They differ very
> little
> > from mine. I just changed the number of workers in rhn.conf,
> but it
> > didn't change anything.
> >
> > I deleted all the channels clones not used by any system and
> dating
> > back from before May 2018, in order to lower the number of
> channels
> > in the queue. There were 127 channels in the queue before these
> > deletion (indicated
> in /var/log/rhn/rhn_taskomatic_daemon.log), and
> > there are 361 of them now ... I must admit I'm confused... I
> hoped
> > it would reduce the number of channels to process and thus "help"
> > taskomatic, but obviously I was wrong.
> >
> > I also noticed that the repodata regeneration seems to work
> fine for
> > existing channels that are not clones, but it is not working
> for new
> > channels that are not clones (and not working for new clones but
> > nothing new here).
> >
> > Has anyone got any other idea (even the tiniest) ?
> >
> > Regards,
> > Florence
> >
> >
> > 2018-07-04 15:21 GMT+02:00 Paul Dias - BCX
> <paul.dias at bcx.co.za <mailto:paul.dias at bcx.co.za>
> > <mailto:paul.dias at bcx.co.za <mailto:paul.dias at bcx.co.za>>>:
> >
> > Hi,____
> >
> > __ __
> >
> > Let me post my settings that I have on my CentOS6 server.
> Can’t
> > remember but I have one or two others, but his is from
> the top
> > of my head.____
> >
> > __ __
> >
> > /etc/rhn/rhn.conf____
> >
> > # Added by paul dias increase number of taskomatic workers
> > 20180620____
> >
> > taskomatic.channel_repodata_workers = 3____
> >
> > taskomatic.java.maxmemory=4096____
> >
> > __ __
> >
> > /etc/sysconfig/tomcat6____
> >
> > JAVA_OPTS="-ea -Xms256m -Xmx512m -Djava.awt.headless=true
> > -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
> > -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024
> > -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC
> > -Dnet.sf.ehcache.skipUpdateCheck=true
> >
> -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory"____
> >
> > __ __
> >
> > /etc/tomcat/server.xml____
> >
> > <!-- Define an AJP 1.3 Connector on port 8009 -->____
> >
> > <Connector port="8009" protocol="AJP/1.3"
> > redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1"
> > maxThreads="256" connectionTimeout="20000"/>____
> >
> > __ __
> >
> > <Connector port="8009" protocol="AJP/1.3"
> > redirectPort="8443" URIEncoding="UTF-8" address="::1"
> > maxThreads="256" connectionTimeout="20000"/>____
> >
> > __ __
> >
> > /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf____
> >
> > # Initial Java Heap Size (in MB)____
> >
> > wrapper.java.initmemory=512____
> >
> > __ __
> >
> > # Maximum Java Heap Size (in MB)____
> >
> > wrapper.java.maxmemory=1512____
> >
> > # Adjusted by paul 20180620____
> >
> > __ __
> >
> > wrapper.ping.timeout=0____
> >
> > # # adjusted paul dias 20180620____
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > Regards,____
> >
> > *Paul Dias____*
> >
> > Technical Consultant____
> >
> > 6^th Floor, 8 Boundary Road____
> >
> > Newlands____
> >
> > Cape Town____
> >
> > 7700____
> >
> > T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>
> <tel:+27%2021%20681%203149>____
> >
> > *Meet your future today.____*
> >
> > *__ __*
> >
> > __BCX______
> >
> > __ __
> >
> > __ __
> >
> > __ __
> >
> > __Social-facebook
> > <https://www.facebook.com/BCXworld>____Social-twitter
> > <https://twitter.com/BCXworld>____Social-linkdin
> > <https://za.linkedin.com/BCX>____Social-youtube
> > <https://www.youtube.com/BCXworld>______
> >
> > __ __
> >
> > __ __
> >
> > This e-mail is subject to the BCX electronic
> communication legal
> > notice, available at:
> > https://www.bcx.co.za/disclaimers____
> >
> > /__ __/
> >
> > /__ __/
> >
> > __ __
> >
> > *From:*Paul Dias - BCX
> > *Sent:* 02 July 2018 06:53 PM
> >
> >
> > *To:* spacewalk-list at redhat.com
> <mailto:spacewalk-list at redhat.com> <mailto:spacewalk-list at redhat.com
> <mailto:spacewalk-list at redhat.com>>
> > *Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely
> > without ever generating repodata____
> >
> > __ __
> >
> > What I have noticed, if you use
> > "spacecmd softchannel_generateyumcache <channel name>"
> and then
> > go to tasks and run single repodata bunch, you will see
> it will
> > actually start and generate your channel cache for you on the
> > channel you used the spacecmd on, this works every time.____
> >
> > __ __
> >
> > But yes the task logs just show repodata bunch running
> forever.____
> >
> > __ __
> >
> > Regards,____
> >
> > *Paul Dias*____
> >
> > 6^th Floor, 8 Boundary Road____
> >
> > Newlands____
> >
> > Cape Town____
> >
> > 7700____
> >
> > T: +27 (0) 21 681 3149 <tel:+27%2021%20681%203149>
> <tel:+27%2021%20681%203149>____
> >
> > __ __
> >
> > *Meet your future today.*____
> >
> > **____
> >
> > BCX____
> >
> > __ __
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*Gerald Vogt <vogt at spamcop.net
> <mailto:vogt at spamcop.net> <mailto:vogt at spamcop.net
> <mailto:vogt at spamcop.net>>>
> > *Sent:* Monday, 02 July 2018 9:45 AM
> > *To:* spacewalk-list at redhat.com
> <mailto:spacewalk-list at redhat.com> <mailto:spacewalk-list at redhat.com
> <mailto:spacewalk-list at redhat.com>>
> > *Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely
> > without ever generating repodata____
> >
> > ____
> >
> > After letting the upgraded server sit for a while it
> seems only
> > a few of
> > the task schedules actually finish. By now, only those tasks
> > show up in
> > in the task engine status page:
> >
> > Changelog Cleanup: 2018-07-01 23:00:00 CEST
> FINISHED
> > Clean Log History: 2018-07-01 23:00:00 CEST
> FINISHED
> > Compare Config Files: 2018-07-01 23:00:00 CEST
> FINISHED
> > Daily Summary Mail: 2018-07-01 23:00:00 CEST
> FINISHED
> > Daily Summary Queue: 2018-07-01 23:00:00 CEST
> FINISHED
> >
> > All the other tasks have disappeared from the list by now.
> >
> > The repo-sync tasks seem to work. New packages appear in the
> > channel.
> > However, the repo build is not running or better it seems
> to never
> > properly finish.
> >
> > If I start it manually, it seems to do its work:
> >
> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
> 08:13:10,584 [Thread-12] INFO
> com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
> single-channel-repodata-bunch-0 scheduled succesfully.
> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
> 08:13:10,636 [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
> 08:13:10,651 [DefaultQuartzScheduler_Worker-8] DEBUG
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: task channel-repodata started
> > > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02
> 08:13:10,793 [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,102 [DefaultQuartzScheduler_Worker-8] DEBUG
> com.redhat.rhn.taskomatic.TaskoJob - channel-repodata
> (single-channel-repodata-bunch-0) ... running
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,103 [DefaultQuartzScheduler_Worker-8] INFO
> com.redhat.rhn.taskomatic.TaskoJob -
> single-channel-repodata-bunch-0: bunch channel-repodata-bunch FINISHED
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,137 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
> Modified Date:2018-06-23 03:48:50 CEST
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,137 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
> Modified Date:2018-07-02 03:45:39 CEST
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,211 [Thread-678] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
> Modified Date:2018-06-23 04:09:51 CEST
> > > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02
> 08:13:11,213 [Thread-678] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
> Modified Date:2018-07-02 03:47:55 CEST
> > > INFO | jvm 1 | 2018/07/02 08:13:19 | 2018-07-02
> 08:13:19,062 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
> new repository metadata for channel 'epel6-centos6-x86_64'(sha1)
> 14401 packages, 11613 errata
> > > INFO | jvm 1 | 2018/07/02 08:13:21 | 2018-07-02
> 08:13:21,193 [Thread-678] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
> new repository metadata for channel 'epel7-centos7-x86_64'(sha1)
> 16282 packages, 10176 errata
> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
> 08:40:12,351 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
> metadata generation for 'epel6-centos6-x86_64' finished in 1613 seconds
> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
> 08:40:12,457 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
> Modified Date:2018-06-19 06:28:57 CEST
> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
> 08:40:12,457 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
> Modified Date:2018-07-02 04:30:05 CEST
> > > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02
> 08:40:12,691 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
> new repository metadata for channel
> 'postgresql96-centos7-x86_64'(sha256) 1032 packages, 0 errata
> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
> 08:41:51,710 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
> metadata generation for 'postgresql96-centos7-x86_64' finished in 98
> seconds
> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
> 08:41:51,803 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File
> Modified Date:2018-06-20 05:08:38 CEST
> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
> 08:41:51,803 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel
> Modified Date:2018-07-02 04:00:00 CEST
> > > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02
> 08:41:51,923 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
> new repository metadata for channel
> 'postgresql10-centos6-x86_64'(sha512) 436 packages, 0 errata
> > > INFO | jvm 1 | 2018/07/02 08:42:26 | 2018-07-02
> 08:42:26,479 [Thread-677] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
> metadata generation for 'postgresql10-centos6-x86_64' finished in 34
> seconds
> > > INFO | jvm 1 | 2018/07/02 08:45:01 | 2018-07-02
> 08:45:01,697 [Thread-678] INFO
> com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Repository
> metadata generation for 'epel7-centos7-x86_64' finished in 1900 seconds
> >
> > yet, the task remains in RUNNING. And for whatever reason it
> > only seems
> > to work some channels. I find a total of 20 repos syncing
> in the
> > logs of
> > the updated server compared to 42 repos syncing in the
> logs of
> > the old.
> > I don't really see the difference between those 20 repos
> syncing
> > and
> > those other 22 not. First I suspected channels with
> custom quartz
> > schedules, but then I found channels in both groups.
> >
> > So I don't know how to troubleshoot this any further. The
> > repodata task
> > which I have started 1,5 hours ago is still at "RUNNING". The
> > channels
> > for which the sync works have been updated. I don't know
> why it
> > is still
> > running. Server load is back down...
> >
> > Thanks,
> >
> > Gerald
> >
> > On 22.06.18 19:12, Gerald Vogt wrote:
> > > I have the same problem after upgrading from 2.6 to 2.8
> on CentOS 6.9. I
> > > have even increased the memory as suggested by that
> link but it makes no
> > > differences. None of the scheduled tasks are running. I
> can run a bunch
> > > manually. But the scheduler doesn't seem to work. Last
> execution times
> > > on the task engine status pages are still at timestamps
> from before the
> > > upgrade. -Gerald
> > >
> > >
> > >
> > > On 22.06.18 14:15, Avi Miller wrote:
> > >> Hi,
> > >>
> > >>> On 22 Jun 2018, at 5:51 pm, Florence Savary
> > >>> <florence.savary.fs at gmail.com
> <mailto:florence.savary.fs at gmail.com>
> > <mailto:florence.savary.fs at gmail.com
> <mailto:florence.savary.fs at gmail.com>>> wrote:
> > >>>
> > >>> When using taskotop, we can see a line for the
> channel-repodata task,
> > >>> we see it is running, but there is never any channel
> displayed in the
> > >>> Channel column. We can also see the task marked as
> running in the
> > >>> Admin tab of the WebUI, but if we let it, it never
> stops. The task
> > >>> runs indefinitely, whithout ever doing anything.
> > >>
> > >> If you've never modified the default memory settings,
> Taskomatic is
> > >> probably running out of memory and task is crashing.
> This is a known
> > >> issue, particularly when you sync large repos.
> > >>
> > >> I would suggest increasing the memory assigned to
> Taskomatic to see if
> > >> that resolves the issue. You will need to restart it
> after making
> > >> these changes:
> > >>
> https://docs.oracle.com/cd/E92593_01/E90695/html/swk24-issues-memory.html
> > >>
> > >> Cheers,
> > >> Avi
> > >>
> > >> --
> > >> Oracle <http://www.oracle.com>
> > >> Avi Miller | Product Management Director | +61 (3)
> 8616 3496 <tel:+61%203%208616%203496> <tel:+61%203%208616%203496>
> > >> Oracle Linux and Virtualization
> > >> 417 St Kilda Road, Melbourne, Victoria 3004 Australia
> > >>
> > >>
> > >> _______________________________________________
> > >> Spacewalk-list mailing list
> > >> Spacewalk-list at redhat.com
> <mailto:Spacewalk-list at redhat.com> <mailto:Spacewalk-list at redhat.com
> <mailto:Spacewalk-list at redhat.com>>
> > >> https://www.redhat.com/mailman/listinfo/spacewalk-list
> > >>
> > >
> > > _______________________________________________
> > > Spacewalk-list mailing list
> > > Spacewalk-list at redhat.com
> <mailto:Spacewalk-list at redhat.com> <mailto:Spacewalk-list at redhat.com
> <mailto:Spacewalk-list at redhat.com>>
> > > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
> > ____
> >
> >
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> <mailto:Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>>
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
> >
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> <mailto:Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>>
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
> >
> >
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com <mailto:Spacewalk-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
More information about the Spacewalk-list
mailing list