[Spacewalk-list] Antwort: Re: Errata Cache bunch running forever on Spacewalk 2.4
Jan Dobes
jdobes at redhat.com
Thu Jan 14 14:52:42 UTC 2016
On 12.1.2016 09:08 Paschedag.Netlution at swr.de wrote:
> Hello again,
>
> still having problems,
>
> I noticed that the errata-cache gets triggerd when a client refreshes
> its repos or searches for updates. So I manually fired a "zypper ref -s"
> or a "apt-get update" on every client connected to this spacewalk
> server. This seems to have fixed "something". Suddenly, the
> "errata-cache" task has finished and I got some backtraces from spacewalk.
>
> But I still have trouble. The UI does not seem to get updated correctly
> (on the "system" page, nearly all systems look "up-to-date". But I know,
> it's not!). But the updates are available to the clients (searching on
> the client).
>
> When I look into the last errata-cache schedules (in Spacewalk "Admin"
> -> "Task schedules" -> "errata-cache-bunch") I can see, that the
> "errata-cache" tasks is often "SKIPPED"
>
> errata-cache _2016-01-12 08:50:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:50:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:49:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:49:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:48:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:48:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:47:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:50:00 MEZ FINISHED
> errata-cache _2016-01-12 08:46:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:46:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:45:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:45:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:44:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:46:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:43:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:43:00 MEZ FINISHED
> errata-cache _2016-01-12 08:43:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:43:00 MEZ FINISHED
> errata-cache _2016-01-12 08:42:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:42:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:41:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:41:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:40:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:40:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:39:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:42:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:38:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:38:00 MEZ FINISHED
> errata-cache _2016-01-12 08:38:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:38:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:37:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:37:00 MEZ FINISHED
> errata-cache _2016-01-12 08:37:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:37:00 MEZ FINISHED
> errata-cache _2016-01-12 08:36:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:36:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:35:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:35:00 MEZ SKIPPED
> errata-cache _2016-01-12 08:34:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:36:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:33:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:33:00 MEZ FINISHED
> errata-cache _2016-01-12 08:33:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:33:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:32:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:32:00 MEZ FINISHED
> errata-cache _2016-01-12 08:32:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:32:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:31:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:31:00 MEZ FINISHED
> errata-cache _2016-01-12 08:31:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:31:00 MEZ FINISHED
> errata-mailer _2016-01-12 08:30:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:30:00 MEZ FINISHED
> errata-cache _2016-01-12 08:30:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:30:00 MEZ FINISHED
> errata-cache _2016-01-12 08:29:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> 2016-01-12 08:29:00 MEZ SKIPPED
>
>
>
> And also I noticed that the "errata-mailer" tasks seems to get only
> started, when the "errata-cache" tasks finished within one minute. (Just
> look above). As soon as the errata-cache tasks runs longer than one
> minute, the "errata-mailer" is not started. I'm also not sure, if
> something went wrong, while I was shutting down the spacewalk server for
> the update.
>
> This is what it looked before (on spacewalk 2.2). I started the update
> on 2016-01-04 abount 08:30.
> errata-cache _2016-01-04 10:10:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 10:10:00 MEZ SKIPPED
> errata-cache _2016-01-04 10:00:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 10:00:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:50:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:50:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:40:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:40:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:30:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:30:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:20:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:20:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:10:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:10:00 MEZ SKIPPED
> errata-cache _2016-01-04 09:00:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 09:00:00 MEZ SKIPPED
> errata-cache _2016-01-04 08:50:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 08:50:00 MEZ SKIPPED
> errata-cache _2016-01-04 08:40:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 08:40:00 MEZ SKIPPED
> errata-cache _2016-01-04 08:30:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-05 08:59:59 MEZ INTERRUPTED
> errata-mailer _2016-01-04 08:30:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 08:29:57 MEZ INTERRUPTED
> errata-mailer _2016-01-04 08:00:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 08:00:00 MEZ FINISHED
> errata-cache _2016-01-04 08:00:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 08:00:00 MEZ FINISHED
> errata-mailer _2016-01-04 07:50:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 07:50:00 MEZ FINISHED
> errata-cache _2016-01-04 07:50:00 MEZ _
> <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> 2016-01-04 07:50:00 MEZ FINISHED
>
>
>
> I would be glad, if someone could give me further hints, where to look
> for the root cause to this.
>
> Kind regards,
> Robert
>
>
> Mit freundlichen Grüßen
>
> Robert Paschedag
> Netlution GmbH
> Landteilstr. 33
> 68163 Mannheim
>
> im Auftrag des
> SWR
> Südwestrundfunk
> Informations- und Kommunikationssysteme
> Neckarstraße 230
> 70190 Stuttgart
>
> Telefon +49 (0)711 /929-12654 oder
> Telefon +49 (0)711 /929-13714
> paschedag.netlution at swr.de
>
> swr.de
>
>
>
>
>
> Von: Robert Paschedag <robert.paschedag at web.de>
> An: Paschedag.Netlution at swr.de
> Kopie: spacewalk-list at redhat.com
> Datum: 08.01.2016 21:53
> Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on
> Spacewalk 2.4
> ------------------------------------------------------------------------
>
>
>
> Hi again,
>
> just some further information.
>
> Because I already activated debug logging of taskomatic, searching for
> server errata cache tasks within the rhn_taskomatic_daemon logfiles with
>
> grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk
> '{print $NF; }' | sort | uniq -c
>
> i get
>
> 14 [1000010274]
> 14 [1000010299]
> 4 [1000010504]
> 14 [1000010512]
> 14 [1000010516]
> 14 [1000010517]
> 14 [1000010518]
> 14 [1000010519]
> 14 [1000010520]
> 14 [1000010521]
> 14 [1000010522]
> 14 [1000010523]
> 14 [1000010524]
> 14 [1000010527]
> 14 [1000010528]
> 14 [1000010529]
> 14 [1000010530]
> 14 [1000010531]
> 14 [1000010532]
> 14 [1000010564]
> 14 [1000010570]
> 15 [1000010573]
> 15 [1000010593]
> 13 [1000010639]
> 14 [1000010640]
> 15 [1000010641]
> 13 [1000010642]
> 14 [1000010647]
> 13 [1000010648]
> 14 [1000010649]
> 15 [1000010650]
> 14 [1000010651]
> 14 [1000010652]
> 6 [1000010672]
> 15 [1000010683]
> 14 [1000010684]
> 14 [1000010685]
> 14 [1000010687]
> 14 [1000010688]
> 14 [1000010689]
> 14 [1000010690]
> 14 [1000010691]
> 14 [1000010692]
> 14 [1000010693]
> 14 [1000010694]
> 15 [1000010695]
> 14 [1000010697]
> 14 [1000010698]
> 2 [1000010702]
> 14 [1000010705]
> 14 [1000010710]
> 2 [1000010715]
>
> On this server, there are more than 400 servers registered, so this
> looks like, something is "hanging" on these servers and not going any
> further (i think).
>
> But right now, I don't know - exactly - where to look for this "error"
> or within which DB table to further track this down.
>
> Regards,
> Robert
>
> Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> > Can anybody give a hint, in which db table to search for these
> errata-cache tasks, that are started over and over again? I just don't
> get it myself right now . It also looks like that it is always the same
> prox. 50 servers that this occurs.
> >
> > Regards
> > Robert
> > Am 07.01.2016 10:11 vorm. schrieb Paschedag.Netlution at swr.de:
> >>
> >> Hi all,
> >>
> >> it looks like one (or more?) jobs have been interrupted, while the
> spacewalk service had shut down for upgrade. I upgraded my spacewalk
> server from 2.2 -> 2.3 -> 2.4.
> >>
> >> Now it looks like the errata-cache bunch tasks is running forever
> and the errata notification mail is not started anymore. I think it gets
> called from errata-cache, when this has finished.
> >>
> >> Errata Cache:
> >> 2016-01-07 10:00:00 MEZ
> >> RUNNING
> >> Errata Notification Mail:
> >> 2016-01-04 08:30:00 MEZ
> >> INTERRUPTED
> >>
> >>
> >> When I look into the database, I can see several tasks in the
> rhntaskqueue but I'm pretty sure, that these tasks do not get cleared
> correctly, so the task is run for every server over and over again. I'm
> not sure! This is just my thinking.
> >>
> >> rhnschema=# select * from rhntaskqueue;
> >> org_id | task_name | task_data | priority | earliest
> >>
> --------+----------------------------+------------+----------+-------------------------------
>
> >> 1 | update_server_errata_cache | 1000010701 | 0 |
> 2016-01-07 10:00:45.963006+01
> >> 1 | update_server_errata_cache | 1000010274 | 0 |
> 2016-01-07 10:00:53.198001+01
> >> 1 | update_server_errata_cache | 1000010527 | 0 |
> 2016-01-07 10:01:03.768592+01
> >> 1 | update_server_errata_cache | 1000010689 | 0 |
> 2016-01-07 10:01:56.915296+01
> >> 1 | update_server_errata_cache | 1000010641 | 0 |
> 2016-01-07 10:02:41.775593+01
> >> 1 | update_server_errata_cache | 1000010687 | 0 |
> 2016-01-07 10:05:01.471365+01
> >> 1 | update_server_errata_cache | 1000010650 | 0 |
> 2016-01-07 10:05:26.492405+01
> >> 1 | update_server_errata_cache | 1000010520 | 0 |
> 2016-01-07 10:06:38.052352+01
> >> (8 Zeilen)
> >>
> >> rhnschema=#
> >>
> >> Any help will be appreciated to debug this further.
> >>
> >> Regards,
> >> Robert
> >>
> >>
> >> Mit freundlichen Grüßen
> >>
> >> Robert Paschedag
> >> Netlution GmbH
> >> Landteilstr. 33
> >> 68163 Mannheim
> >>
> >> im Auftrag des
> >> SWR
> >> Südwestrundfunk
> >> Informations- und Kommunikationssysteme
> >> Neckarstraße 230
> >> 70190 Stuttgart
> >>
> >> Telefon +49 (0)711 /929-12654 oder
> >> Telefon +49 (0)711 /929-13714
> >> paschedag.netlution at swr.de
> >>
> >> swr.de
> >>
> >
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
>
>
>
Hello,
since Spacewalk 2.3 this type of task is running every minute instead of
every ten minutes before. However any of these tasks should not take so
long. These tasks are picked by taskomatic daemon and processed. Are
there any exceptions in /var/log/rhn/rhn_taskomatic_daemon.log ? Try to
restart taskomatic.
Regards,
--
Jan Dobes
Satellite Engineering, Red Hat
More information about the Spacewalk-list
mailing list