[Spacewalk-list] Taskomatic runs indefinitely without ever generating repodata
Matt Moldvan
matt at moldvan.com
Thu Jul 5 14:05:46 UTC 2018
How is the server utilization with respect to disk I/O (something like
iotop or htop might help here)? Maybe there is something else blocking and
the server doesn't have enough resources to complete. Have you tried
running an strace against the running process?
I also had an (well, many) issue(s) with our Spacewalk server before
disabling snapshots in /etc/rhn/rhn.conf. I also increased the number of
workers and max repodata work items:
# system snapshots enabled
enable_snapshots = 0
...
taskomatic.maxmemory=6144
taskomatic.errata_cache_max_work_items = 500
taskomatic.channel_repodata_max_work_items = 50
taskomatic.channel_repodata_workers = 5
On Thu, Jul 5, 2018 at 4:38 AM Florence Savary <florence.savary.fs at gmail.com>
wrote:
> Hello,
>
> Thanks for sharing your configuration files. They differ very little from
> mine. I just changed the number of workers in rhn.conf, but it didn't
> change anything.
>
> I deleted all the channels clones not used by any system and dating back
> from before May 2018, in order to lower the number of channels in the
> queue. There were 127 channels in the queue before these deletion
> (indicated in /var/log/rhn/rhn_taskomatic_daemon.log), and there are 361 of
> them now ... I must admit I'm confused... I hoped it would reduce the
> number of channels to process and thus "help" taskomatic, but obviously I
> was wrong.
>
> I also noticed that the repodata regeneration seems to work fine for
> existing channels that are not clones, but it is not working for new
> channels that are not clones (and not working for new clones but nothing
> new here).
>
> Has anyone got any other idea (even the tiniest) ?
>
> Regards,
> Florence
>
>
> 2018-07-04 15:21 GMT+02:00 Paul Dias - BCX <paul.dias at bcx.co.za>:
>
>> Hi,
>>
>>
>>
>> Let me post my settings that I have on my CentOS6 server. Can’t remember
>> but I have one or two others, but his is from the top of my head.
>>
>>
>>
>> /etc/rhn/rhn.conf
>>
>> # Added by paul dias increase number of taskomatic workers 20180620
>>
>> taskomatic.channel_repodata_workers = 3
>>
>> taskomatic.java.maxmemory=4096
>>
>>
>>
>> /etc/sysconfig/tomcat6
>>
>> JAVA_OPTS="-ea -Xms256m -Xmx512m -Djava.awt.headless=true
>> -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
>> -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX:MaxNewSize=256
>> -XX:-UseConcMarkSweepGC -Dnet.sf.ehcache.skipUpdateCheck=true
>> -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory"
>>
>>
>>
>> /etc/tomcat/server.xml
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>> URIEncoding="UTF-8" address="127.0.0.1" maxThreads="256"
>> connectionTimeout="20000"/>
>>
>>
>>
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>> URIEncoding="UTF-8" address="::1" maxThreads="256"
>> connectionTimeout="20000"/>
>>
>>
>>
>> /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf
>>
>> # Initial Java Heap Size (in MB)
>>
>> wrapper.java.initmemory=512
>>
>>
>>
>> # Maximum Java Heap Size (in MB)
>>
>> wrapper.java.maxmemory=1512
>>
>> # Adjusted by paul 20180620
>>
>>
>>
>> wrapper.ping.timeout=0
>>
>> # # adjusted paul dias 20180620
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> *Paul Dias*
>>
>> Technical Consultant
>>
>> 6th Floor, 8 Boundary Road
>>
>> Newlands
>>
>> Cape Town
>>
>> 7700
>>
>> T: +27 (0) 21 681 3149 <+27%2021%20681%203149>
>>
>> *Meet your future today.*
>>
>>
>>
>> [image: BCX]
>>
>>
>>
>>
>>
>>
>>
>> [image: Social-facebook] <https://www.facebook.com/BCXworld>[image:
>> Social-twitter] <https://twitter.com/BCXworld>[image: Social-linkdin]
>> <https://za.linkedin.com/BCX>[image: Social-youtube]
>> <https://www.youtube.com/BCXworld>
>>
>>
>>
>>
>>
>> This e-mail is subject to the BCX electronic communication legal notice,
>> available at:
>> https://www.bcx.co.za/disclaimers
>>
>>
>>
>>
>>
>>
>>
>> *From:* Paul Dias - BCX
>> *Sent:* 02 July 2018 06:53 PM
>>
>> *To:* spacewalk-list at redhat.com
>> *Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely without
>> ever generating repodata
>>
>>
>>
>> What I have noticed, if you use "spacecmd softchannel_generateyumcache
>> <channel name>" and then go to tasks and run single repodata bunch, you
>> will see it will actually start and generate your channel cache for you on
>> the channel you used the spacecmd on, this works every time.
>>
>>
>>
>> But yes the task logs just show repodata bunch running forever.
>>
>>
>>
>> Regards,
>>
>> *Paul Dias*
>>
>> 6th Floor, 8 Boundary Road
>>
>> Newlands
>>
>> Cape Town
>>
>> 7700
>>
>> T: +27 (0) 21 681 3149 <+27%2021%20681%203149>
>>
>>
>>
>> *Meet your future today.*
>>
>>
>>
>> [image: BCX]
>>
>>
>> ------------------------------
>>
>> *From:* Gerald Vogt <vogt at spamcop.net>
>> *Sent:* Monday, 02 July 2018 9:45 AM
>> *To:* spacewalk-list at redhat.com
>> *Subject:* Re: [Spacewalk-list] Taskomatic runs indefinitely without
>> ever generating repodata
>>
>>
>>
>> After letting the upgraded server sit for a while it seems only a few of
>> the task schedules actually finish. By now, only those tasks show up in
>> in the task engine status page:
>>
>> Changelog Cleanup: 2018-07-01 23:00:00 CEST FINISHED
>> Clean Log History: 2018-07-01 23:00:00 CEST FINISHED
>> Compare Config Files: 2018-07-01 23:00:00 CEST FINISHED
>> Daily Summary Mail: 2018-07-01 23:00:00 CEST FINISHED
>> Daily Summary Queue: 2018-07-01 23:00:00 CEST FINISHED
>>
>> All the other tasks have disappeared from the list by now.
>>
>> The repo-sync tasks seem to work. New packages appear in the channel.
>> However, the repo build is not running or better it seems to never
>> properly finish.
>>
>> If I start it manually, it seems to do its work:
>>
>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,584
>> [Thread-12] INFO com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job
>> single-channel-repodata-bunch-0 scheduled succesfully.
>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,636
>> [DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob
>> - single-channel-repodata-bunch-0: bunch channel-repodata-bunch STARTED
>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,651
>> [DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob
>> - single-channel-repodata-bunch-0: task channel-repodata started
>> > INFO | jvm 1 | 2018/07/02 08:13:10 | 2018-07-02 08:13:10,793
>> [DefaultQuartzScheduler_Worker-8] INFO
>> com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 4
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,102
>> [DefaultQuartzScheduler_Worker-8] DEBUG com.redhat.rhn.taskomatic.TaskoJob
>> - channel-repodata (single-channel-repodata-bunch-0) ... running
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,103
>> [DefaultQuartzScheduler_Worker-8] INFO com.redhat.rhn.taskomatic.TaskoJob
>> - single-channel-repodata-bunch-0: bunch channel-repodata-bunch FINISHED
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,137
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> File Modified Date:2018-06-23 03:48:50 CEST
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,137
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Channel Modified Date:2018-07-02 03:45:39 CEST
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,211
>> [Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> File Modified Date:2018-06-23 04:09:51 CEST
>> > INFO | jvm 1 | 2018/07/02 08:13:11 | 2018-07-02 08:13:11,213
>> [Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Channel Modified Date:2018-07-02 03:47:55 CEST
>> > INFO | jvm 1 | 2018/07/02 08:13:19 | 2018-07-02 08:13:19,062
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Generating new repository metadata for channel 'epel6-centos6-x86_64'(sha1)
>> 14401 packages, 11613 errata
>> > INFO | jvm 1 | 2018/07/02 08:13:21 | 2018-07-02 08:13:21,193
>> [Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Generating new repository metadata for channel 'epel7-centos7-x86_64'(sha1)
>> 16282 packages, 10176 errata
>> > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,351
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Repository metadata generation for 'epel6-centos6-x86_64' finished in 1613
>> seconds
>> > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,457
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> File Modified Date:2018-06-19 06:28:57 CEST
>> > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,457
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Channel Modified Date:2018-07-02 04:30:05 CEST
>> > INFO | jvm 1 | 2018/07/02 08:40:12 | 2018-07-02 08:40:12,691
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Generating new repository metadata for channel
>> 'postgresql96-centos7-x86_64'(sha256) 1032 packages, 0 errata
>> > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,710
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Repository metadata generation for 'postgresql96-centos7-x86_64' finished
>> in 98 seconds
>> > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,803
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> File Modified Date:2018-06-20 05:08:38 CEST
>> > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,803
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Channel Modified Date:2018-07-02 04:00:00 CEST
>> > INFO | jvm 1 | 2018/07/02 08:41:51 | 2018-07-02 08:41:51,923
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Generating new repository metadata for channel
>> 'postgresql10-centos6-x86_64'(sha512) 436 packages, 0 errata
>> > INFO | jvm 1 | 2018/07/02 08:42:26 | 2018-07-02 08:42:26,479
>> [Thread-677] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Repository metadata generation for 'postgresql10-centos6-x86_64' finished
>> in 34 seconds
>> > INFO | jvm 1 | 2018/07/02 08:45:01 | 2018-07-02 08:45:01,697
>> [Thread-678] INFO com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter -
>> Repository metadata generation for 'epel7-centos7-x86_64' finished in 1900
>> seconds
>>
>> yet, the task remains in RUNNING. And for whatever reason it only seems
>> to work some channels. I find a total of 20 repos syncing in the logs of
>> the updated server compared to 42 repos syncing in the logs of the old.
>> I don't really see the difference between those 20 repos syncing and
>> those other 22 not. First I suspected channels with custom quartz
>> schedules, but then I found channels in both groups.
>>
>> So I don't know how to troubleshoot this any further. The repodata task
>> which I have started 1,5 hours ago is still at "RUNNING". The channels
>> for which the sync works have been updated. I don't know why it is still
>> running. Server load is back down...
>>
>> Thanks,
>>
>> Gerald
>>
>> On 22.06.18 19:12, Gerald Vogt wrote:
>> > I have the same problem after upgrading from 2.6 to 2.8 on CentOS 6.9.
>> I
>> > have even increased the memory as suggested by that link but it makes
>> no
>> > differences. None of the scheduled tasks are running. I can run a bunch
>> > manually. But the scheduler doesn't seem to work. Last execution times
>> > on the task engine status pages are still at timestamps from before the
>> > upgrade. -Gerald
>> >
>> >
>> >
>> > On 22.06.18 14:15, Avi Miller wrote:
>> >> Hi,
>> >>
>> >>> On 22 Jun 2018, at 5:51 pm, Florence Savary
>> >>> <florence.savary.fs at gmail.com> wrote:
>> >>>
>> >>> When using taskotop, we can see a line for the channel-repodata task,
>> >>> we see it is running, but there is never any channel displayed in the
>> >>> Channel column. We can also see the task marked as running in the
>> >>> Admin tab of the WebUI, but if we let it, it never stops. The task
>> >>> runs indefinitely, whithout ever doing anything.
>> >>
>> >> If you've never modified the default memory settings, Taskomatic is
>> >> probably running out of memory and task is crashing. This is a known
>> >> issue, particularly when you sync large repos.
>> >>
>> >> I would suggest increasing the memory assigned to Taskomatic to see if
>> >> that resolves the issue. You will need to restart it after making
>> >> these changes:
>> >>
>> https://docs.oracle.com/cd/E92593_01/E90695/html/swk24-issues-memory.html
>> >>
>> >> Cheers,
>> >> Avi
>> >>
>> >> --
>> >> Oracle <http://www.oracle.com>
>> >> Avi Miller | Product Management Director | +61 (3) 8616 3496
>> <+61%203%208616%203496>
>> >> Oracle Linux and Virtualization
>> >> 417 St Kilda Road, Melbourne, Victoria 3004 Australia
>> >>
>> >>
>> >> _______________________________________________
>> >> Spacewalk-list mailing list
>> >> Spacewalk-list at redhat.com
>> >> https://www.redhat.com/mailman/listinfo/spacewalk-list
>> >>
>> >
>> > _______________________________________________
>> > Spacewalk-list mailing list
>> > Spacewalk-list at redhat.com
>> > https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.jpg
Type: image/jpeg
Size: 1700 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.jpg
Type: image/jpeg
Size: 478 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.jpg
Type: image/jpeg
Size: 1586 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 2866 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 1625 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20180705/bc9d5aa5/attachment-0004.jpg>
More information about the Spacewalk-list
mailing list