[Spacewalk-list] Antwort: Re: Antwort: Re: Errata Cache bunch running forever on Spacewalk 2.4
Paschedag.Netlution at swr.de
Paschedag.Netlution at swr.de
Mon Jan 18 12:39:03 UTC 2016
Hi Jan,
ok. Maybe, my problem is fixed now. This might be, because I really - had
no other idea - to remove the whole "errata-cache-bunch" schedule for a
short time to be sure, that nothing gets scheduled anymore. Then, I
recreated the "errata-cache-default" schedule again.
Now, sometimes, the errata-cache tasks takes longer than one minute to
complete. Maybe this is totally normal and nobody ever recognized it
before spacewalk 2.3, because the schedule was every 10 minutes, not as
you said, every minute right now.
But I'm still wondering, why the "errata-mailer" task does NOT get
startet, if the errata-cache tasks does NOT finish within that one minute.
Can anybody confirm this behavior?
Again an example
errata-mailer 2016-01-18 13:17:00 MEZ 2016-01-18 13:17:00 MEZ
FINISHED
errata-cache 2016-01-18 13:17:00 MEZ 2016-01-18 13:17:00 MEZ
FINISHED
#### errata-mailer MISSING (from my understanding). Triggered by task
startet at 13:15:00 MEZ
errata-cache 2016-01-18 13:16:00 MEZ 2016-01-18 13:16:00 MEZ
SKIPPED # skipped because task still running
errata-cache 2016-01-18 13:15:00 MEZ 2016-01-18 13:16:00 MEZ
FINISHED # Task took longer than one minute
Kind regards,
Robert
Mit freundlichen Grüßen
Robert Paschedag
Netlution GmbH
Landteilstr. 33
68163 Mannheim
im Auftrag des
SWR
Südwestrundfunk
Informations- und Kommunikationssysteme
Neckarstraße 230
70190 Stuttgart
Telefon +49 (0)711 /929-12654 oder
Telefon +49 (0)711 /929-13714
paschedag.netlution at swr.de
swr.de
Von: Robert Paschedag <robert.paschedag at web.de>
An: Jan Dobes <jdobes at redhat.com>
Kopie: spacewalk-list at redhat.com
Datum: 14.01.2016 20:53
Betreff: Re: [Spacewalk-list] Antwort: Re: Errata Cache bunch
running forever on Spacewalk 2.4
Gesendet von: spacewalk-list-bounces at redhat.com
Hello Jan,
I don't see anything useful, if debug logging is disabled in taskomatic.
Also when debugging is disabled, the only thing I see, is, that some
errata-cache tasks are "SKIPPED". I also enabled all logging of SQL
statements in postgres but - IMHO - everything looks " good".
I already started the whole server, restarted taskomatic, restarted
spacewalk-service. Nothing seems to help.
I'm not sure, if sometimes the threads are colliding, so I wanted to set
the number of workers to 1 for some time.
On my test system, everything looks good. But there are only about 10
systems registered, so I can't tell, if this "might" be some kind of
performance problem.
Regards
Robert
Am 14.01.2016 3:52 nachm. schrieb Jan Dobes <jdobes at redhat.com>:
>
> On 12.1.2016 09:08 Paschedag.Netlution at swr.de wrote:
> > Hello again,
> >
> > still having problems,
> >
> > I noticed that the errata-cache gets triggerd when a client refreshes
> > its repos or searches for updates. So I manually fired a "zypper ref
-s"
> > or a "apt-get update" on every client connected to this spacewalk
> > server. This seems to have fixed "something". Suddenly, the
> > "errata-cache" task has finished and I got some backtraces from
spacewalk.
> >
> > But I still have trouble. The UI does not seem to get updated
correctly
> > (on the "system" page, nearly all systems look "up-to-date". But I
know,
> > it's not!). But the updates are available to the clients (searching on
> > the client).
> >
> > When I look into the last errata-cache schedules (in Spacewalk "Admin"
> > -> "Task schedules" -> "errata-cache-bunch") I can see, that the
> > "errata-cache" tasks is often "SKIPPED"
> >
> > errata-cache _2016-01-12 08:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:50:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:49:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:49:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:48:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:48:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:47:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:50:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:46:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:46:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:45:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:45:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:44:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:46:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:43:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:43:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:43:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:43:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:42:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:42:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:41:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:41:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:40:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:39:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:42:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:38:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:38:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:38:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:38:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:37:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:37:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:37:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:37:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:36:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:36:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:35:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:35:00 MEZ SKIPPED
> > errata-cache _2016-01-12 08:34:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:36:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:33:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:33:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:33:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:33:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:32:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:32:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:32:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:32:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:31:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:31:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:31:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:31:00 MEZ FINISHED
> > errata-mailer _2016-01-12 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:30:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:30:00 MEZ FINISHED
> > errata-cache _2016-01-12 08:29:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132>
> > 2016-01-12 08:29:00 MEZ SKIPPED
> >
> >
> >
> > And also I noticed that the "errata-mailer" tasks seems to get only
> > started, when the "errata-cache" tasks finished within one minute.
(Just
> > look above). As soon as the errata-cache tasks runs longer than one
> > minute, the "errata-mailer" is not started. I'm also not sure, if
> > something went wrong, while I was shutting down the spacewalk server
for
> > the update.
> >
> > This is what it looked before (on spacewalk 2.2). I started the update
> > on 2016-01-04 abount 08:30.
> > errata-cache _2016-01-04 10:10:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 10:10:00 MEZ SKIPPED
> > errata-cache _2016-01-04 10:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 10:00:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:50:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:40:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:30:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:20:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:20:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:10:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:10:00 MEZ SKIPPED
> > errata-cache _2016-01-04 09:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 09:00:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:50:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:40:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:40:00 MEZ SKIPPED
> > errata-cache _2016-01-04 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-05 08:59:59 MEZ INTERRUPTED
> > errata-mailer _2016-01-04 08:30:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:29:57 MEZ INTERRUPTED
> > errata-mailer _2016-01-04 08:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:00:00 MEZ FINISHED
> > errata-cache _2016-01-04 08:00:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 08:00:00 MEZ FINISHED
> > errata-mailer _2016-01-04 07:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 07:50:00 MEZ FINISHED
> > errata-cache _2016-01-04 07:50:00 MEZ _
> > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7>
> > 2016-01-04 07:50:00 MEZ FINISHED
> >
> >
> >
> > I would be glad, if someone could give me further hints, where to look
> > for the root cause to this.
> >
> > Kind regards,
> > Robert
> >
> >
> > Mit freundlichen Grüßen
> >
> > Robert Paschedag
> > Netlution GmbH
> > Landteilstr. 33
> > 68163 Mannheim
> >
> > im Auftrag des
> > SWR
> > Südwestrundfunk
> > Informations- und Kommunikationssysteme
> > Neckarstraße 230
> > 70190 Stuttgart
> >
> > Telefon +49 (0)711 /929-12654 oder
> > Telefon +49 (0)711 /929-13714
> > paschedag.netlution at swr.de
> >
> > swr.de
> >
> >
> >
> >
> >
> > Von: Robert Paschedag <robert.paschedag at web.de>
> > An: Paschedag.Netlution at swr.de
> > Kopie: spacewalk-list at redhat.com
> > Datum: 08.01.2016 21:53
> > Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on
> > Spacewalk 2.4
> >
------------------------------------------------------------------------
> >
> >
> >
> > Hi again,
> >
> > just some further information.
> >
> > Because I already activated debug logging of taskomatic, searching for
> > server errata cache tasks within the rhn_taskomatic_daemon logfiles
with
> >
> > grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* |
awk
> > '{print $NF; }' | sort | uniq -c
> >
> > i get
> >
> > 14 [1000010274]
> > 14 [1000010299]
> > 4 [1000010504]
> > 14 [1000010512]
> > 14 [1000010516]
> > 14 [1000010517]
> > 14 [1000010518]
> > 14 [1000010519]
> > 14 [1000010520]
> > 14 [1000010521]
> > 14 [1000010522]
> > 14 [1000010523]
> > 14 [1000010524]
> > 14 [1000010527]
> > 14 [1000010528]
> > 14 [1000010529]
> > 14 [1000010530]
> > 14 [1000010531]
> > 14 [1000010532]
> > 14 [1000010564]
> > 14 [1000010570]
> > 15 [1000010573]
> > 15 [1000010593]
> > 13 [1000010639]
> > 14 [1000010640]
> > 15 [1000010641]
> > 13 [1000010642]
> > 14 [1000010647]
> > 13 [1000010648]
> > 14 [1000010649]
> > 15 [1000010650]
> > 14 [1000010651]
> > 14 [1000010652]
> > 6 [1000010672]
> > 15 [1000010683]
> > 14 [1000010684]
> > 14 [1000010685]
> > 14 [1000010687]
> > 14 [1000010688]
> > 14 [1000010689]
> > 14 [1000010690]
> > 14 [1000010691]
> > 14 [1000010692]
> > 14 [1000010693]
> > 14 [1000010694]
> > 15 [1000010695]
> > 14 [1000010697]
> > 14 [1000010698]
> > 2 [1000010702]
> > 14 [1000010705]
> > 14 [1000010710]
> > 2 [1000010715]
> >
> > On this server, there are more than 400 servers registered, so this
> > looks like, something is "hanging" on these servers and not going any
> > further (i think).
> >
> > But right now, I don't know - exactly - where to look for this "error"
> > or within which DB table to further track this down.
> >
> > Regards,
> > Robert
> >
> > Am 08.01.2016 um 19:31 schrieb Robert Paschedag:
> > > Can anybody give a hint, in which db table to search for these
> > errata-cache tasks, that are started over and over again? I just don't
> > get it myself right now . It also looks like that it is always the
same
> > prox. 50 servers that this occurs.
> > >
> > > Regards
> > > Robert
> > > Am 07.01.2016 10:11 vorm. schrieb Paschedag.Netlution at swr.de:
> > >>
> > >> Hi all,
> > >>
> > >> it looks like one (or more?) jobs have been interrupted, while the
> > spacewalk service had shut down for upgrade. I upgraded my spacewalk
> > server from 2.2 -> 2.3 -> 2.4.
> > >>
> > >> Now it looks like the errata-cache bunch tasks is running forever
> > and the errata notification mail is not started anymore. I think it
gets
> > called from errata-cache, when this has finished.
> > >>
> > >> Errata Cache:
> > >> 2016-01-07 10:00:00 MEZ
> > >> RUNNING
> > >> Errata Notification Mail:
> > >> 2016-01-04 08:30:00 MEZ
> > >> INTERRUPTED
> > >>
> > >>
> > >> When I look into the database, I can see several tasks in the
> > rhntaskqueue but I'm pretty sure, that these tasks do not get cleared
> > correctly, so the task is run for every server over and over again.
I'm
> > not sure! This is just my thinking.
> > >>
> > >> rhnschema=# select * from rhntaskqueue;
> > >> org_id | task_name | task_data | priority |
earliest
> > >>
> >
--------+----------------------------+------------+----------+-------------------------------
> >
> > >> 1 | update_server_errata_cache | 1000010701 | 0 |
> > 2016-01-07 10:00:45.963006+01
> > >> 1 | update_server_errata_cache | 1000010274 | 0 |
> > 2016-01-07 10:00:53.198001+01
> > >> 1 | update_server_errata_cache | 1000010527 | 0 |
> > 2016-01-07 10:01:03.768592+01
> > >> 1 | update_server_errata_cache | 1000010689 | 0 |
> > 2016-01-07 10:01:56.915296+01
> > >> 1 | update_server_errata_cache | 1000010641 | 0 |
> > 2016-01-07 10:02:41.775593+01
> > >> 1 | update_server_errata_cache | 1000010687 | 0 |
> > 2016-01-07 10:05:01.471365+01
> > >> 1 | update_server_errata_cache | 1000010650 | 0 |
> > 2016-01-07 10:05:26.492405+01
> > >> 1 | update_server_errata_cache | 1000010520 | 0 |
> > 2016-01-07 10:06:38.052352+01
> > >> (8 Zeilen)
> > >>
> > >> rhnschema=#
> > >>
> > >> Any help will be appreciated to debug this further.
> > >>
> > >> Regards,
> > >> Robert
> > >>
> > >>
> > >> Mit freundlichen Grüßen
> > >>
> > >> Robert Paschedag
> > >> Netlution GmbH
> > >> Landteilstr. 33
> > >> 68163 Mannheim
> > >>
> > >> im Auftrag des
> > >> SWR
> > >> Südwestrundfunk
> > >> Informations- und Kommunikationssysteme
> > >> Neckarstraße 230
> > >> 70190 Stuttgart
> > >>
> > >> Telefon +49 (0)711 /929-12654 oder
> > >> Telefon +49 (0)711 /929-13714
> > >> paschedag.netlution at swr.de
> > >>
> > >> swr.de
> > >>
> > >
> > > _______________________________________________
> > > Spacewalk-list mailing list
> > > Spacewalk-list at redhat.com
> > > https://www.redhat.com/mailman/listinfo/spacewalk-list
> > >
> >
> >
> >
>
> Hello,
>
> since Spacewalk 2.3 this type of task is running every minute instead of
> every ten minutes before. However any of these tasks should not take so
> long. These tasks are picked by taskomatic daemon and processed. Are
> there any exceptions in /var/log/rhn/rhn_taskomatic_daemon.log ? Try to
> restart taskomatic.
>
> Regards,
> --
> Jan Dobes
> Satellite Engineering, Red Hat
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160118/148dbbad/attachment.htm>
More information about the Spacewalk-list
mailing list