[Spacewalk-list] Spacewalk 2.8 clients not checking in

Tue Jul 24 23:36:26 UTC 2018

I'll chime in and add to Jason Lewis and Nicholas Penner's observations.
We run a mix of CentOS and various LTS releases of Ubuntu on our Spacewalk
environment. Last week I figured out that the CentOS Spacewalk clients
weren't being updated because the channel/repo I had setup was only for
the Spacewalk backend. I therefore set up a new channel/repo for the
Spacewalk 2.8 client and added all our CentOS servers to it so they would
get upgraded. That's what happened last weekend and since all our CentOS
servers now fail to check in.

1) because our Ubuntu clients don't support OSAD or config deployments, I
never bothered to get OSAD/jabber stable for the smaller number of CentOS
servers
2) the CentOS clients were checking in fine with the original 2.6 and 2.7
clients but stopped working when upgraded to 2.8
3) all our Ubuntu clients still check in regularly

So the issue appears to be with the CentOS rhnsd service scheduling
rhn_checks, not the backend.
rhnsd is not a script, it's an executable, so "fixing the script" isn't
really an option.

Presumably, for people successfully using OSAD, checks still happen as a
result of OSAD, and so the failure of rhnsd to check in isn't being
noticed. Alternatively (I didn't look into the code) perhaps the scheduled
task (through taskomatic or does rhnsd use its own scheduler?) somehow
tries to do an OSAD check (or something else that fails) before running
rhn_check, and when the OSAD check fails then aborts silently prior to
actually running rhn_check?

Anyways, this is a regression with the Centos 2.8 client that's a
departure from 2.7 behaviour and, since this mechanism appears to be a
failsafe for OSAD failures or lack of implementation, it's no longer
working as designed.

What changes were made to rhnsd in 2.8 that might cause this regression?

------------------------------

Message: 2
Date: Thu, 19 Jul 2018 19:48:48 +0000
From: "Jason W. Lewis" <jwlewis at microcenter.com>
To: "spacewalk-list at redhat.com" <spacewalk-list at redhat.com>
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in
Message-ID: <25DAA02E-3533-4469-99D0-29CE3B03CCE4 at microcenter.com>
Content-Type: text/plain; charset="utf-8"

Now I?m more confused.  Our Spacewalk environment has never had OSA
installed on the client side.  (I inherited this, btw.).  Our old clients
are still checking in just fine without it, just like they always have.
So?

  1.  Is OSA really necessary (it looks like it?s only needed when a
client hasn?t checked in for a while)?
  2.  Rhn_check still isn?t working on the 2.8 clients, so configuring (or
in our case, installing) OSA doesn?t seem like a resolution to the actual
problem; it will (again, if I?m reading this right) only cover up the
symptoms.

I can schedule cron jobs to run rhn_check manually, which will make this
work, but I?d rather get the problem solved.  Any additional thoughts on
what could be the source of the problem?
--
Jason Lewis
Senior Systems Engineer @Micro Electronics, Inc.
P: 614.777.2728 | M:
jwlewis at microcenter.com<mailto:jwlewis at microcenter.com>

From: <spacewalk-list-bounces at redhat.com> on behalf of "Penner, Nicholas
(HSAL)" <Nicholas.Penner at gov.mb.ca>
Reply-To: "spacewalk-list at redhat.com" <spacewalk-list at redhat.com>
Date: Thursday, July 19, 2018 at 12:04
To: "spacewalk-list at redhat.com" <spacewalk-list at redhat.com>
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in

**Attention**
The following message was received from outside the Micro Electronics
network. Carefully examine it before you open any links or attachments.

--------------------------
I?ve done that a few times now and to no avail. What?s weird is restarting
rhnsd on the client makes it check in fine. Also restarting the actual
spacewalk server makes the agents with issues check in.

That being said, OSAD is working fine as even when an agent hasn?t checked
in for over a day it can still do actions on those hosts through spacewalk
and they are picked up immediately.

So not 100% sure why that is but I?ve tried doing what you mentioned,
tried messing with rhnsd on the client side to adjust the check time and
nadda.

Regards,

Nicholas Penner
Technical Specialist
Manitoba Health
3041-300 Carlton Street
Winnipeg, MB, R3B 3M9
204-788-6498
nicholas.penner at gov.mb.ca<mailto:curtis.brown at gov.mb.ca>

Confidentiality Notice: This message and any attachment to it are intended
for the addressee only and may contain legally privileged or confidential
information. Any unauthorized use, disclosure, distribution, or copying is
strictly prohibited. Please notify the sender if you have received this
E-mail by mistake, and please delete it and the attachments (and all
copies) in a secure manner.  Thank You.
Message de confidentialit? : Ce message et tout document dans cette
transmission est destin? ? la personne ou aux personnes ? qui il est
adress?. Il peut contenir des informations privil?gi?es ou
confidentielles. Toute utilisation, divulgation, distribution ou copie non
autoris?e est strictement d?fendue. Si vous n'?tes pas le destinataire de
ce courriel, veuillez en informer l'exp?diteur et effacer l'original (et
toutes les pi?ces jointes) de mani?re s?curitaire. Merci.

From: spacewalk-list-bounces at redhat.com
<spacewalk-list-bounces at redhat.com> On Behalf Of Dimitri Yioulos
Sent: July 19, 2018 9:59 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in

When you say you upgraded some clients, assuming you?re using osad, it may
be that you need to do some refreshing.  I?d check
/var/lib/jabberd/db/sqlite.db on your Spacewalk server to make sure it?s
not a zero-length file.  Then, I?d stop Spacewalk services, delete the
sqlite database (rm -rf /var/lib/jabberd/db/*), restart Spacewalk
services, then check the sqlite.db again.  On the clients, you should stop
osad, remove /etc/sysconfig/rhn/osad-auth.conf, then start osad.

Hth.

Dimitri

From:
spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com
>
<spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.co
m>> On Behalf Of Penner, Nicholas (HSAL)
Sent: Thursday, July 19, 2018 10:16 AM
To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in

I get the same as what Jason said. For us it?s not all hosts, only some.

The hosts that are still running 2.7 client work fine as well.

Regards,

Nicholas Penner
Technical Specialist
Manitoba Health
3041-300 Carlton Street
Winnipeg, MB, R3B 3M9
204-788-6498
nicholas.penner at gov.mb.ca<mailto:curtis.brown at gov.mb.ca>

Confidentiality Notice: This message and any attachment to it are intended
for the addressee only and may contain legally privileged or confidential
information. Any unauthorized use, disclosure, distribution, or copying is
strictly prohibited. Please notify the sender if you have received this
E-mail by mistake, and please delete it and the attachments (and all
copies) in a secure manner.  Thank You.
Message de confidentialit? : Ce message et tout document dans cette
transmission est destin? ? la personne ou aux personnes ? qui il est
adress?. Il peut contenir des informations privil?gi?es ou
confidentielles. Toute utilisation, divulgation, distribution ou copie non
autoris?e est strictement d?fendue. Si vous n'?tes pas le destinataire de
ce courriel, veuillez en informer l'exp?diteur et effacer l'original (et
toutes les pi?ces jointes) de mani?re s?curitaire. Merci.

From:
spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com
>
<spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.co
m>> On Behalf Of Jason W. Lewis
Sent: July 19, 2018 8:59 AM
To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in

Robert,

Thanks for the input, but I?ve already looked at rhnsd, and the interval
variable is set.  I?ve even tried adjusting it, but rhn_check isn?t even
starting, so it makes no difference.

--
Jason Lewis
Senior Systems Engineer @Micro Electronics, Inc.
P: 614.777.2728 | M:
jwlewis at microcenter.com<mailto:jwlewis at microcenter.com>

From:
<spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.co
m>> on behalf of William Hongach
<William.Hongach at marist.edu<mailto:William.Hongach at marist.edu>>
Reply-To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>"
<spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Date: Wednesday, July 18, 2018 at 16:57
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>"
<spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: Re: [Spacewalk-list] Spacewalk 2.8 clients not checking in

**Attention**
The following message was received from outside the Micro Electronics
network. Carefully examine it before you open any links or attachments.

--------------------------
Hi Jason,

The check-in interval is controlled by the INTERVAL variable in
/etc/sysconfig/rhn/rhnsd.  By default, I believe it is set to INTERVAL=240
(4 hrs).  Have you checked this value on the affected clients to determine
if it is behaving as configured?

I believe rhnsd will -not- check in at reboot, unless you configure it to
do so (rhn_check in crontab, etc.).  It will check in at rhnsd_start_time
+ interval.  This is the behavior I have observed with 2.8.

Hope this helps.

From:
spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.com
>
<spacewalk-list-bounces at redhat.com<mailto:spacewalk-list-bounces at redhat.co
m>> On Behalf Of Jason W. Lewis
Sent: Wednesday, July 18, 2018 4:36 PM
To: spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>
Subject: [Spacewalk-list] Spacewalk 2.8 clients not checking in

Hi All,

I just upgraded to 2.8 (server and some clients) and now the clients that
have been upgraded aren?t checking in anymore.  I?ve tried restarting the
rhnsd service, even rebooted the system, and still nothing.  I?ve looked
at /var/log/up2date, and there?s no activity since the time rhnsd was last
started.  It looks like rhn_check just isn?t getting run.  I ran it
manually and the system checked in and picked up the chain I had waiting
for it.

Is anyone familiar with this enough to provide a solution, or at least
know enough to help me get started troubleshooting this?

Thank you,
--
Jason Lewis
Senior Systems Engineer @Micro Electronics, Inc.
P: 614.777.2728 | M:
jwlewis at microcenter.com<mailto:jwlewis at microcenter.com>