[Spacewalk-list] Antwort: Re: Errata Cache bunch running forever on Spacewalk 2.4
Robert Paschedag
robert.paschedag at web.de
Thu Jan 14 19:40:17 UTC 2016
Hello Jan,
I don't see anything useful, if debug logging is disabled in taskomatic.
Also when debugging is disabled, the only thing I see, is, that some errata-cache tasks are "SKIPPED". I also enabled all logging of SQL statements in postgres but - IMHO - everything looks " good".
I already started the whole server, restarted taskomatic, restarted spacewalk-service. Nothing seems to help.
I'm not sure, if sometimes the threads are colliding, so I wanted to set the number of workers to 1 for some time.
On my test system, everything looks good. But there are only about 10 systems registered, so I can't tell, if this "might" be some kind of performance problem.
Regards
Robert
Am 14.01.2016 3:52 nachm. schrieb Jan Dobes <jdobes at redhat.com>:
>
> On 12.1.2016 09:08 Paschedag.Netlution at swr.de wrote:
> > Hello again,
> >
> > still having problems,
> >
> > I noticed that the errata-cache gets triggerd when a client refreshes
> > its repos or searches for updates. So I manually fired a "zypper ref -s"
> > or a "apt-get update" on every client connected to this spacewalk
> > server. This seems to have fixed "something". Suddenly, the
> > "errata-cache" task has finished and I got some backtraces from spacewalk.
> >
> > But I still have trouble. The UI does not seem to get updated correctly
> > (on the "system" page, nearly all systems look "up-to-date". But I know,
> > it's not!). But the updates are available to the clients (searching on
> > the client).
> >
> > When I look into the last errata-cache schedules (in Spacewalk "Admin"
> > -> "Task schedules" -> "errata-cache-bunch") I can see, that the
> > "errata-cache" tasks is often "SKIPPED"
> >
> > errata-cache _2016-01-12 08:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:50:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:49:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:49:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:48:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:48:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:47:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:50:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:46:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:46:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:45:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:45:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:44:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:46:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:43:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:43:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:43:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:43:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:42:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:42:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:41:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:41:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:40:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:39:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:42:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:38:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:38:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:38:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:38:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:37:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:37:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:37:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:37:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:36:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:36:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:35:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:35:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:34:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:36:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:33:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:33:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:33:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:33:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:32:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:32:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:32:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:32:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:31:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:31:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:31:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:31:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:30:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:30:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:29:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:29:00 MEZ SKIPPED
> >
> >
> >
> > And also I noticed that the "errata-mailer" tasks seems to get only
> > started, when the "errata-cache" tasks finished within one minute. (Just
> > look above). As soon as the errata-cache tasks runs longer than one
> > minute, the "errata-mailer" is not started. I'm also not sure, if
> > something went wrong, while I was shutting down the spacewalk server for
> > the update.
> >
> > This is what it looked before (on spacewalk 2.2). I started the update
> > on 2016-01-04 abount 08:30.
> > errata-cache _2016-01-04 10:10:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 10:10:00 MEZ SKIPPED
> > errata-cache _2016-01-04 10:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 10:00:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:50:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:40:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:30:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:20:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:20:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:10:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:10:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:00:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:50:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:40:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-05 08:59:59 MEZ INTERRUPTED
> > errata-mailer _2016-01-04 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:29:57 MEZ INTERRUPTED
> > errata-mailer _2016-01-04 08:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:00:00 MEZ FINISHED
> > errata-cache _2016-01-04 08:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:00:00 MEZ FINISHED
> > errata-mailer _2016-01-04 07:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 07:50:00 MEZ FINISHED
> > errata-cache _2016-01-04 07:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 07:50:00 MEZ FINISHED
> >
> >
> >
> > I would be glad, if someone could give me further hints, where to look
> > for the root cause to this.
> >
> > Kind regards,
> > Robert
> >
> >
> > Mit freundlichen Grüßen
> >
> > Robert Paschedag
> > Netlution GmbH
> > Landteilstr. 33
> > 68163 Mannheim
> >
> > im Auftrag des
> > SWR
> > Südwestrundfunk
> > Informations- und Kommunikationssysteme
> > Neckarstraße 230
> > 70190 Stuttgart
> >
> > Telefon +49 (0)711 /929-12654 oder
> > Telefon +49 (0)711 /929-13714
> > paschedag.netlution at swr.de
> >
> > swr.de
> >
> >
> >
> >
> >
> > Von: Robert Paschedag <robert.paschedag at web.de>
> > An: Paschedag.Netlution at swr.de
> > Kopie: spacewalk-list at redhat.com
> > Datum: 08.01.2016 21:53
> > Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on
> > Spacewalk 2.4
> > ------------------------------------------------------------------------
> >
> >
> >
> > Hi again,
> >
> > just some further information.
> >
> > Because I already activated debug logging of taskomatic, searching for
> > server errata cache tasks within the rhn_taskomatic_daemon logfiles with
> >
> > grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk
> > '{print $NF; }' | sort | uniq -c
> >
> > i get
> >
> > 14 [1000010274]
> > 14 [1000010299]
> > 4 [1000010504]
> > 14 [1000010512]
> > 14 [1000010516]
> > 14 [1000010517]
> > 14 [1000010518]
> > 14 [1000010519]
> > 14 [1000010520]
> > 14 [1000010521]
> > 14 [1000010522]
> > 14 [1000010523]
> > 14 [1000010524]
> > 14 [1000010527]
> > 14 [1000010528]
> > 14 [1000010529]
> > 14 [1000010530]
> > 14 [1000010531]
> > 14 [1000010532]
> > 14 [1000010564]
> > 14 [1000010570]
> > 15 [1000010573]
> > 15 [1000010593]
> > 13 [1000010639]
> > 14 [1000010640]
> > 15 [1000010641]
> > 13 [1000010642]
> > 14 [1000010647]
> > 13 [1000010648]
> > 14 [1000010649]
> > 15 [1000010650]
> > 14 [1000010651]
> > 14 [1000010652]
> > 6 [1000010672]
> > 15 [1000010683]
> > 14 [1000010684]
> > 14 [1000010685]
> > 14 [1000010687]
> > 14 [1000010688]
> > 14 [1000010689]
> > 14 [1000010690]
> > 14 [1000010691]
> > 14 [1000010692]
> > 14 [1000010693]
> > 14 [1000010694]
> > 15 [1000010695]
> > 14 [1000010697]
> > 14 [1000010698]
> > 2 [1000010702]
> > 14 [1000010705]
> > 14 [1000010710]
> > 2 [1000010715]
> >
> > On this server, there are more than 400 servers registered, so this
> > looks like, something is "hanging" on these servers and not going any
> > further (i think).
> >
> > But right now, I don't know - exactly - where to look for this "error"
> > or within which DB table to further track this down.
> >
> > Regards,
> > Robert
> >
> > Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> > > Can anybody give a hint, in which db table to search for these
> > errata-cache tasks, that are started over and over again? I just don't
> > get it myself right now . It also looks like that it is always the same
> > prox. 50 servers that this occurs.
> > >
> > > Regards
> > > Robert
> > > Am 07.01.2016 10:11 vorm. schrieb Paschedag.Netlution at swr.de:
> > >>
> > >> Hi all,
> > >>
> > >> it looks like one (or more?) jobs have been interrupted, while the
> > spacewalk service had shut down for upgrade. I upgraded my spacewalk
> > server from 2.2 -> 2.3 -> 2.4.
> > >>
> > >> Now it looks like the errata-cache bunch tasks is running forever
> > and the errata notification mail is not started anymore. I think it gets
> > called from errata-cache, when this has finished.
> > >>
> > >> Errata Cache:
> > >> 2016-01-07 10:00:00 MEZ
> > >> RUNNING
> > >> Errata Notification Mail:
> > >> 2016-01-04 08:30:00 MEZ
> > >> INTERRUPTED
> > >>
> > >>
> > >> When I look into the database, I can see several tasks in the
> > rhntaskqueue but I'm pretty sure, that these tasks do not get cleared
> > correctly, so the task is run for every server over and over again. I'm
> > not sure! This is just my thinking.
> > >>
> > >> rhnschema=# select * from rhntaskqueue;
> > >> org_id | task_name | task_data | priority | earliest
> > >>
> > --------+----------------------------+------------+----------+-------------------------------
> >
> > >> 1 | update_server_errata_cache | 1000010701 | 0 |
> > 2016-01-07 10:00:45.963006+01
> > >> 1 | update_server_errata_cache | 1000010274 | 0 |
> > 2016-01-07 10:00:53.198001+01
> > >> 1 | update_server_errata_cache | 1000010527 | 0 |
> > 2016-01-07 10:01:03.768592+01
> > >> 1 | update_server_errata_cache | 1000010689 | 0 |
> > 2016-01-07 10:01:56.915296+01
> > >> 1 | update_server_errata_cache | 1000010641 | 0 |
> > 2016-01-07 10:02:41.775593+01
> > >> 1 | update_server_errata_cache | 1000010687 | 0 |
> > 2016-01-07 10:05:01.471365+01
> > >> 1 | update_server_errata_cache | 1000010650 | 0 |
> > 2016-01-07 10:05:26.492405+01
> > >> 1 | update_server_errata_cache | 1000010520 | 0 |
> > 2016-01-07 10:06:38.052352+01
> > >> (8 Zeilen)
> > >>
> > >> rhnschema=#
> > >>
> > >> Any help will be appreciated to debug this further.
> > >>
> > >> Regards,
> > >> Robert
> > >>
> > >>
> > >> Mit freundlichen Grüßen
> > >>
> > >> Robert Paschedag
> > >> Netlution GmbH
> > >> Landteilstr. 33
> > >> 68163 Mannheim
> > >>
> > >> im Auftrag des
> > >> SWR
> > >> Südwestrundfunk
> > >> Informations- und Kommunikationssysteme
> > >> Neckarstraße 230
> > >> 70190 Stuttgart
> > >>
> > >> Telefon +49 (0)711 /929-12654 oder
> > >> Telefon +49 (0)711 /929-13714
> > >> paschedag.netlution at swr.de
> > >>
> > >> swr.de
> > >>
> > >
> > > _______________________________________________
> > > Spacewalk-list mailing list
> > > Spacewalk-list at redhat.com
> > > https://www.redhat.com/mailman/listinfo/spacewalk-list
> > >
> >
> >
> >
>
> Hello,
>
> since Spacewalk 2.3 this type of task is running every minute instead of
> every ten minutes before. However any of these tasks should not take so
> long. These tasks are picked by taskomatic daemon and processed. Are
> there any exceptions in /var/log/rhn/rhn_taskomatic_daemon.log ? Try to
> restart taskomatic.
>
> Regards,
> --
> Jan Dobes
> Satellite Engineering, Red Hat
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
More information about the Spacewalk-list
mailing list