[Spacewalk-list] Spacewalk 2.10 Failing to sync repo
Jody McIvor
JMcIvor at bclc.com
Tue Feb 18 15:31:37 UTC 2020
Hi Michael,
Disregard previous, our dev env nightly build wasn't updating nightly. Manual push of .26 resolved the issue. Thanks a ton!
Jody McIvor
Sr. Systems Administrator, Integration
Les Services D'intégration
[TEL] (250) 852-5202
jmcivor at bclc.com
BCLC.com
-----Original Message-----
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> On Behalf Of spacewalk-list-request at redhat.com
Sent: February 14, 2020 9:16 AM
To: spacewalk-list at redhat.com
Subject: Spacewalk-list Digest, Vol 141, Issue 19
Send Spacewalk-list mailing list submissions to
spacewalk-list at redhat.com
To subscribe or unsubscribe via the World Wide Web, visit
https://www.redhat.com/mailman/listinfo/spacewalk-list
or, via email, send a message with subject or body 'help' to
spacewalk-list-request at redhat.com
You can reach the person managing the list at
spacewalk-list-owner at redhat.com
When replying, please edit your Subject line so it is more specific than "Re: Contents of Spacewalk-list digest..."
Today's Topics:
1. Spacewalk 2.10 Failing to sync repo (Michael Mraka) (Jody McIvor)
2. Re: "Invalid function call attempted (code 6)" (Paul Greene)
----------------------------------------------------------------------
Message: 1
Date: Fri, 14 Feb 2020 17:03:25 +0000
From: Jody McIvor <JMcIvor at bclc.com>
To: "spacewalk-list at redhat.com" <spacewalk-list at redhat.com>
Subject: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo (Michael
Mraka)
Message-ID: <7730df3361b44aab9ffbf705daadb1e1 at bclc.com>
Content-Type: text/plain; charset=WINDOWS-1252
Hi Michael,
I ran your suggested command to peek at the header info of one of (most recent) the RPMs:
====================================================
[jmcivor@<MYREPOSERVER>]$ rpm -qip <PACKAGENAME>.rpm
Name : <PACKAGENAME>-v2.15.01
Epoch : 0
Version : 1
Release : 1
Architecture: x86_64
Install Date: (not installed)
Group : BCLC
Size : 8121032
License : 2017 BCLC
Signature : (none)
Source RPM : terminal-source
Build Date : Thu 06 Feb 2020 01:44:20 PM PST Build Host : <BUILDHOST> Relocations : (not relocatable)
Packager : <USERNAMEOFPACKAGER>
Vendor : BCLC
URL : n/a
Summary : terminal
Description :
<APPDESCRIPTION>
====================================================
And ran manual repo sync:
====================================================
[root@<SPACEWALKSERVER>]# spacewalk-repo-sync --channel <CHANNELNAME> --type yum
08:42:02 ======================================
08:42:02 | Channel: <CHANNELNAME>
08:42:02 ======================================
08:42:02 Sync of channel started.
08:42:02
08:42:02 Processing repository with URL: http://<REPOSERVER>/rhel/x86_64/<CHANNELNAME> /
08:42:03 Packages in repo: 111
08:42:03 Packages already synced: 65
08:42:03 Packages to sync: 14
08:42:03 New packages to download: 14
08:42:03 Downloading packages:
08:42:03 1/14 : <PACKAGENAME>.rpm
08:42:03 2/14 : <PACKAGENAME>.rpm
08:42:03 3/14 : <PACKAGENAME>.rpm
08:42:03 4/14 : <PACKAGENAME>.rpm
08:42:03 5/14 : <PACKAGENAME>.rpm
08:42:03 6/14 : <PACKAGENAME>.rpm
08:42:04 7/14 : <PACKAGENAME>.rpm
08:42:04 8/14 : <PACKAGENAME>.rpm
08:42:04 9/14 : <PACKAGENAME>.rpm
08:42:04 10/14 : <PACKAGENAME>.rpm
08:42:04 11/14 : <PACKAGENAME>.rpm
08:42:04 12/14 : <PACKAGENAME>.rpm
08:42:04 13/14 : <PACKAGENAME>.rpm
08:42:04 14/14 : <PACKAGENAME>.rpm
08:42:05
08:42:05 Importing packages to DB:
Importing packages: |##################################################| 100.0%
08:42:05
08:42:05 Errata in repo: 0.
08:42:05 Sync of channel completed in 0:00:03.
08:42:05 Total time: 0:00:03
====================================================
As usual, it reports completion, but digging into the actual log for this channels reposync (var/log/rhn/reposync/<CHANNELNAME>.log) gives us the details I provided previously:
====================================================
2020/02/14 08:42:05 -07:00 Sync of channel completed in 0:00:03.
2020/02/14 08:48:08 -07:00 Command: ['/bin/spacewalk-repo-sync', '--channel', '<CHANNELNAME>', '--type', 'yum']
2020/02/14 08:48:08 -07:00 Sync of channel started.
2020/02/14 08:48:08 -07:00
2020/02/14 08:48:08 -07:00 Processing repository with URL: http://<REPOSERVER>/rhel/x86_64/<CHANNELNAME>/
2020/02/14 08:48:08 -07:00 Packages in repo: 111
2020/02/14 08:48:08 -07:00 Packages already synced: 65
2020/02/14 08:48:08 -07:00 Packages to sync: 14
2020/02/14 08:48:08 -07:00 New packages to download: 14
2020/02/14 08:48:08 -07:00 Downloading packages:
2020/02/14 08:48:09 -07:00 1/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 2/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 3/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 4/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 5/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 6/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 7/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 8/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 9/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 10/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 11/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 12/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 13/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:09 -07:00 14/14 : <PACKAGENAME>.rpm
2020/02/14 08:48:10 -07:00 Importing packages started.
2020/02/14 08:48:10 -07:00
2020/02/14 08:48:10 -07:00 Importing packages to DB:
2020/02/14 08:48:10 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 unknown header tag
2020/02/14 08:48:11 -07:00 Importing packages finished.
2020/02/14 08:48:11 -07:00
2020/02/14 08:48:11 -07:00 Errata in repo: 0.
2020/02/14 08:48:11 -07:00 Sync of channel completed in 0:00:03.
====================================================
If I don't hear from you today, have a great V-day :)
Thanks,
Jody McIvor
Sr. Systems Administrator, Integration
Les Services D'int?gration
[TEL] (250) 852-5202
jmcivor at bclc.com
BCLC.com
-----Original Message-----
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> On Behalf Of spacewalk-list-request at redhat.com
Sent: February 14, 2020 1:06 AM
To: spacewalk-list at redhat.com
Subject: Spacewalk-list Digest, Vol 141, Issue 16
Send Spacewalk-list mailing list submissions to spacewalk-list at redhat.com
To subscribe or unsubscribe via the World Wide Web, visit https://www.redhat.com/mailman/listinfo/spacewalk-list
or, via email, send a message with subject or body 'help' to spacewalk-list-request at redhat.com
You can reach the person managing the list at spacewalk-list-owner at redhat.com
When replying, please edit your Subject line so it is more specific than "Re: Contents of Spacewalk-list digest..."
Today's Topics:
1. "Invalid function call attempted (code 6)" (Paul Greene)
2. Re: "Invalid function call attempted (code 6)" (Paul Greene)
3. Re: "Invalid function call attempted (code 6)" (Ezequiel Sozzi)
4. Re: Spacewalk 2.10 Failing to sync repo (Michael Mraka)
----------------------------------------------------------------------
Message: 1
Date: Thu, 13 Feb 2020 17:25:30 -0500
From: Paul Greene <paul.greene.va at gmail.com>
To: spacewalk-list at redhat.com
Subject: [Spacewalk-list] "Invalid function call attempted (code 6)"
Message-ID:
<CAMAkkvPzCUBjJPo8JrphSsboxSUA4fkaw8T9oWL3X35qTT03yA at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I have a spacewalk 2.9 server with CentOS 7 clients. When I run a scheduled remote command on 50 systems, usually about half of the systems will get marked as "failed" with the error "Invalid function call attempted (code 6)".
They all have the same configuration, and every line put in the remote command will run just fine from a command prompt. If I go into a system that has been marked "failed" and manually verify if the command did what it was supposed to do, many times it actually did succeed, but was still marked "failed". And there are some that did in fact fail.
How can I address this error to get rid of the false "failed" messages?
I looked in /var/log/up2date on the clients that failed and get just these messages at the time the scheduled task failed:
up2date updateLoginfo() login info
up2date logging into up2date server
up2date successfully retrieved authentication token from up2date server
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/cee09dea/attachment.html>
------------------------------
Message: 2
Date: Thu, 13 Feb 2020 18:26:16 -0500
From: Paul Greene <paul.greene.va at gmail.com>
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] "Invalid function call attempted (code 6)"
Message-ID:
<CAMAkkvORhbxcqe2E+U5JTc5HHyynT9wCYLO_EPwqYoDkonQyoQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Just to add some more info that might be relevant.
The spacewalk server itself is running CentOS 7.7, kernel 3.10.0-1062.9.1.el7.x86_64, and most of the clients that are failing are running the same kernel.
There's some machines running an older kernel - 3.10.0-862.14.4.el7.x86_64
- and on these machines they seem to reliably run the task without problems.
A small number of machines on the newer kernel succeed with the task, but most don't.
On Thu, Feb 13, 2020 at 5:25 PM Paul Greene <paul.greene.va at gmail.com>
wrote:
> I have a spacewalk 2.9 server with CentOS 7 clients. When I run a
> scheduled remote command on 50 systems, usually about half of the
> systems will get marked as "failed" with the error "Invalid function
> call attempted (code 6)".
>
> They all have the same configuration, and every line put in the remote
> command will run just fine from a command prompt. If I go into a
> system that has been marked "failed" and manually verify if the
> command did what it was supposed to do, many times it actually did
> succeed, but was still marked "failed". And there are some that did in fact fail.
>
> How can I address this error to get rid of the false "failed" messages?
>
> I looked in /var/log/up2date on the clients that failed and get just
> these messages at the time the scheduled task failed:
>
> up2date updateLoginfo() login info
> up2date logging into up2date server
> up2date successfully retrieved authentication token from up2date
> server
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/d301b339/attachment.html>
------------------------------
Message: 3
Date: Thu, 13 Feb 2020 21:22:23 -0300
From: Ezequiel Sozzi <sozeze at gmail.com>
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] "Invalid function call attempted (code 6)"
Message-ID:
<CALqkNvL0M+zPnuZW-FtuKYc9M82sQHtFTm4208NRaai25VhA_w at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Paul,
This a more often issue than everybody things, in order to fix this, what we do is to run the next commands on the client side:
Disable all the plugins to disable rhnplugin: sed -i 's/plugins=1/plugins=0/g' /etc/yum.conf
Disable all the external repositores: yum-config-manager --disable \*
Re-enable all the plugins to enable rhnplugin: sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf Update all the packages related to rpm, rhn, and yum: yum update rpm* rhn*
yum* -y
This fix the issue. At least that's my experience. Hope this helps.
BR,
Ezequiel
El jue., 13 de febrero de 2020 7:26 p. m., Paul Greene < paul.greene.va at gmail.com> escribi?:
> I have a spacewalk 2.9 server with CentOS 7 clients. When I run a
> scheduled remote command on 50 systems, usually about half of the
> systems will get marked as "failed" with the error "Invalid function
> call attempted (code 6)".
>
> They all have the same configuration, and every line put in the remote
> command will run just fine from a command prompt. If I go into a
> system that has been marked "failed" and manually verify if the
> command did what it was supposed to do, many times it actually did
> succeed, but was still marked "failed". And there are some that did in fact fail.
>
> How can I address this error to get rid of the false "failed" messages?
>
> I looked in /var/log/up2date on the clients that failed and get just
> these messages at the time the scheduled task failed:
>
> up2date updateLoginfo() login info
> up2date logging into up2date server
> up2date successfully retrieved authentication token from up2date
> server _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200213/aa87e3fe/attachment.html>
------------------------------
Message: 4
Date: Fri, 14 Feb 2020 10:06:01 +0100
From: Michael Mraka <michael.mraka at redhat.com>
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo
Message-ID: <20200214090601.GD3694 at magni.z31.cz>
Content-Type: text/plain; charset=utf-8
Jody McIvor:
...
> 2020/02/13 08:09:02 -07:00 Command: ['/usr/bin/spacewalk-repo-sync',
> '--channel', <MY CHANNEL NAME>', '--type', 'yum']
> 2020/02/13 08:09:02 -07:00 Sync of channel started.
> 2020/02/13 08:09:02 -07:00
> 2020/02/13 08:09:02 -07:00 Processing repository with URL: http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/
> 2020/02/13 08:09:03 -07:00 Packages in repo: 111
> 2020/02/13 08:09:04 -07:00 Packages already synced: 65
> 2020/02/13 08:09:04 -07:00 Packages to sync: 14
> 2020/02/13 08:09:04 -07:00 New packages to download: 14
> 2020/02/13 08:09:04 -07:00 Downloading packages:
> 2020/02/13 08:09:04 -07:00 1/14 : <PACKAGENAME>.rpm
...
> 2020/02/13 08:09:08 -07:00 Importing packages to DB:
> 2020/02/13 08:09:08 -07:00 unknown header tag
Hello,
I guess rpm packages you try to sync contain a new header which is not recognized by rpm on your server.
Try to run from commandline (on server)
# rpm -qip <PACKAGENAME>.rpm
# /usr/bin/spacewalk-repo-sync --channel <MY CHANNEL NAME> and watch the errors.
Regards,
--
Michael Mr?ka
System Management Engineering, Red Hat
------------------------------
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
End of Spacewalk-list Digest, Vol 141, Issue 16
***********************************************
________________________________
This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email.
------------------------------
Message: 2
Date: Fri, 14 Feb 2020 12:15:17 -0500
From: Paul Greene <paul.greene.va at gmail.com>
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] "Invalid function call attempted (code
6)"
Message-ID:
<CAMAkkvOWeDkQDWBkU7wTUzuCb6=GkgGy=iyGQAM==NAcB8Achg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Ezequiel,
When I run rhn_check -vvvv, the only line in there that looks like an error message is "couldn't find any keys in /var/lib/rpm/pubkeys/*.key", (the pubkey folder doesn't exist)
but then the next line is "loading keyring from rpmdb", so it looks like it gets the key from there.
I get those same messages though on both the machines that fail and the ones that succeed.
Paul
On Fri, Feb 14, 2020 at 11:51 AM Paul Greene <paul.greene.va at gmail.com>
wrote:
> What version of 7 are you running? (including kernel #?)
>
> On Fri, Feb 14, 2020 at 11:17 AM Ezequiel Sozzi <sozeze at gmail.com> wrote:
>
>> Hi Paul,
>>
>> Versions should not be the problem, I'm managing almost 4000 servers
>> with spacewalk and 35% are Centos6 while the other 65% are Centos7.
>> have you tried to perform a rhn_check -vvvv from the client? That
>> could bring you more information.
>>
>> BR,
>>
>> El vie., 14 de feb. de 2020 a la(s) 13:12, Paul Greene (
>> paul.greene.va at gmail.com) escribi?:
>>
>>> Ezequiel,
>>>
>>> I tried it but it didn't seem to do anything. ?
>>> These systems have no connection to the internet - our repositories
>>> are all internal to the network (one repo for base, one for updates,
>>> and one for EPEL), and they have all the latest updates anyway, so
>>> there was nothing to update.
>>> Not sure where to go with this.
>>> Just to add to my second post - older versions of CentOS 7 aren't
>>> having issues, and there's many systems still on CentOS 6 that don't
>>> have any issues either. So that leads me to believe there's
>>> something about the differences in OS versions that are the root of the problem.
>>>
>>> Paul
>>>
>>> On Thu, Feb 13, 2020 at 7:23 PM Ezequiel Sozzi <sozeze at gmail.com> wrote:
>>>
>>>> Hi Paul,
>>>>
>>>> This a more often issue than everybody things, in order to fix
>>>> this, what we do is to run the next commands on the client side:
>>>>
>>>> Disable all the plugins to disable rhnplugin: sed -i
>>>> 's/plugins=1/plugins=0/g' /etc/yum.conf
>>>>
>>>> Disable all the external repositores: yum-config-manager --disable
>>>> \*
>>>>
>>>> Re-enable all the plugins to enable rhnplugin: sed -i
>>>> 's/plugins=0/plugins=1/g' /etc/yum.conf Update all the packages
>>>> related to rpm, rhn, and yum: yum update rpm*
>>>> rhn* yum* -y
>>>>
>>>> This fix the issue. At least that's my experience. Hope this helps.
>>>>
>>>> BR,
>>>>
>>>> Ezequiel
>>>>
>>>>
>>>>
>>>> El jue., 13 de febrero de 2020 7:26 p. m., Paul Greene <
>>>> paul.greene.va at gmail.com> escribi?:
>>>>
>>>>> I have a spacewalk 2.9 server with CentOS 7 clients. When I run a
>>>>> scheduled remote command on 50 systems, usually about half of the
>>>>> systems will get marked as "failed" with the error "Invalid
>>>>> function call attempted (code 6)".
>>>>>
>>>>> They all have the same configuration, and every line put in the
>>>>> remote command will run just fine from a command prompt. If I go
>>>>> into a system that has been marked "failed" and manually verify if
>>>>> the command did what it was supposed to do, many times it actually
>>>>> did succeed, but was still marked "failed". And there are some that did in fact fail.
>>>>>
>>>>> How can I address this error to get rid of the false "failed" messages?
>>>>>
>>>>> I looked in /var/log/up2date on the clients that failed and get
>>>>> just these messages at the time the scheduled task failed:
>>>>>
>>>>> up2date updateLoginfo() login info up2date logging into up2date
>>>>> server up2date successfully retrieved authentication token from
>>>>> up2date server _______________________________________________
>>>>> Spacewalk-list mailing list
>>>>> Spacewalk-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>>
>>>> _______________________________________________
>>>> Spacewalk-list mailing list
>>>> Spacewalk-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>>
>>> _______________________________________________
>>> Spacewalk-list mailing list
>>> Spacewalk-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>>
>> _______________________________________________
>> Spacewalk-list mailing list
>> Spacewalk-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/spacewalk-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.redhat.com/archives/spacewalk-list/attachments/20200214/07d90de0/attachment.html>
------------------------------
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
End of Spacewalk-list Digest, Vol 141, Issue 19
***********************************************
________________________________
This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email.
More information about the Spacewalk-list
mailing list