[Spacewalk-list] Spacewalk 2.10 Failing to sync repo
Jody McIvor
JMcIvor at bclc.com
Thu Feb 13 16:32:16 UTC 2020
Thank you so much for giving this some attention and patching, Michael!
Sadly, the issue remains. First thing this morning I logged into the instance of Spacewalk experiencing this issue and ensured that it had indeed received spacewalk-backend-tools-2.10.25 overnight, and it had. So I immediately attempted a repository sync and at first glance it seems all the same errors are still occurring. Oddly enough, Error code 14 seems to not exist (I've googled the heck out of it a few times in recent months/weeks). Attempting to sync a channel that was created a few weeks ago during testing returns error 5.
- Email rcvd from spacewalk taskomatic:
"Taskomatic bunch repo-sync-bunch was scheduled to run within the single-repo-sync-bunch-1 schedule. Subtask repo-sync failed. For more information check /var/log/rhn/tasko/org1/repo-sync-bunch/repo-sync_9902924_err."
- Contents of repo-sync_9902924_err:
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an exception: org.quartz.JobExecutionException
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command '[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
2020-02-13 08:09:09,050 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack trace:org.quartz.JobExecutionException: Command '[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
at com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
at com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
-Contents of <MY CHANNEL NAME>.log:
2020/02/13 08:09:02 -07:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', <MY CHANNEL NAME>', '--type', 'yum']
2020/02/13 08:09:02 -07:00 Sync of channel started.
2020/02/13 08:09:02 -07:00
2020/02/13 08:09:02 -07:00 Processing repository with URL: http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/
2020/02/13 08:09:03 -07:00 Packages in repo: 111
2020/02/13 08:09:04 -07:00 Packages already synced: 65
2020/02/13 08:09:04 -07:00 Packages to sync: 14
2020/02/13 08:09:04 -07:00 New packages to download: 14
2020/02/13 08:09:04 -07:00 Downloading packages:
2020/02/13 08:09:04 -07:00 1/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 2/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 3/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:04 -07:00 4/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:05 -07:00 5/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:05 -07:00 6/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 7/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 8/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:06 -07:00 9/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:07 -07:00 10/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:07 -07:00 11/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 12/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 13/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 14/14 : <PACKAGENAME>.rpm
2020/02/13 08:09:08 -07:00 Importing packages started.
2020/02/13 08:09:08 -07:00
2020/02/13 08:09:08 -07:00 Importing packages to DB:
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 unknown header tag
2020/02/13 08:09:08 -07:00 Importing packages finished.
2020/02/13 08:09:08 -07:00
2020/02/13 08:09:08 -07:00 Errata in repo: 0.
2020/02/13 08:09:08 -07:00 Sync of channel completed in 0:00:06.
-Contents of rhn_taskomatic_daemon.log:
INFO | jvm 1 | 2020/02/13 08:09:01 | 2020-02-13 08:09:01,753 [DefaultQuartzScheduler_Worker-3] INFO com.redhat.rhn.taskomatic.task.RepoSyncTask - Syncing repos for channel: <MY CHANNEL NAME>
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,029 [Thread-51] INFO com.redhat.rhn.taskomatic.task.RepoSyncTask - 08:09:02 ======================================
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 | Channel: <MY CHANNEL NAME>
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 ======================================
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 Sync of channel started.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:02 Processing repository with URL: http://<MYREPOSERVER>/rhel/x86_64/<MYREPOFOLDER/
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:03 Packages in repo: 111
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Packages already synced: 65
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Packages to sync: 14
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 New packages to download: 14
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 Downloading packages:
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 1/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 2/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 3/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:04 4/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:05 5/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:05 6/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 7/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 8/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:06 9/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:07 10/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:07 11/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 12/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 13/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 14/14 : <PACKAGENAME>.rpm
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Importing packages to DB:
INFO | jvm 1 | 2020/02/13 08:09:09 |
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |####----------------------------------------------| 7.14%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |#######-------------------------------------------| 14.29%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |###########---------------------------------------| 21.43%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |##############------------------------------------| 28.57%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |##################--------------------------------| 35.71%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |#####################-----------------------------| 42.86%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |#########################-------------------------| 50.0%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |#############################---------------------| 57.14%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |################################------------------| 64.29%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |####################################--------------| 71.43%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |#######################################-----------| 78.57%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |###########################################-------| 85.71%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |##############################################----| 92.86%
INFO | jvm 1 | 2020/02/13 08:09:09 | Importing packages: |##################################################| 100.0%
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Errata in repo: 0.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Sync of channel completed in 0:00:06.
INFO | jvm 1 | 2020/02/13 08:09:09 | 08:09:08 Total time: 0:00:06
INFO | jvm 1 | 2020/02/13 08:09:09 |
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an exception: org.quartz.JobExecutionException
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command '[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,034 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
INFO | jvm 1 | 2020/02/13 08:09:09 | 2020-02-13 08:09:09,050 [DefaultQuartzScheduler_Worker-3] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack trace:org.quartz.JobExecutionException: Command '[/usr/bin/spacewalk-repo-sync, --channel, <MY CHANNEL NAME>, --type, yum]' exited with error code 14
INFO | jvm 1 | 2020/02/13 08:09:09 | at com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
INFO | jvm 1 | 2020/02/13 08:09:09 | at com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
INFO | jvm 1 | 2020/02/13 08:09:09 | at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
INFO | jvm 1 | 2020/02/13 08:09:09 | at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
INFO | jvm 1 | 2020/02/13 08:09:09 | at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO | jvm 1 | 2020/02/13 08:09:09 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Jody McIvor
Sr. Systems Administrator, Integration
Les Services D'intégration
[TEL] (250) 852-5202
jmcivor at bclc.com
BCLC.com
-----Original Message-----
From: spacewalk-list-bounces at redhat.com <spacewalk-list-bounces at redhat.com> On Behalf Of spacewalk-list-request at redhat.com
Sent: February 12, 2020 9:00 AM
To: spacewalk-list at redhat.com
Subject: Spacewalk-list Digest, Vol 141, Issue 14
Send Spacewalk-list mailing list submissions to
spacewalk-list at redhat.com
To subscribe or unsubscribe via the World Wide Web, visit
https://www.redhat.com/mailman/listinfo/spacewalk-list
or, via email, send a message with subject or body 'help' to
spacewalk-list-request at redhat.com
You can reach the person managing the list at
spacewalk-list-owner at redhat.com
When replying, please edit your Subject line so it is more specific than "Re: Contents of Spacewalk-list digest..."
Today's Topics:
1. Re: Spacewalk 2.10 Failing to sync repo (Michael Mraka)
----------------------------------------------------------------------
Message: 1
Date: Wed, 12 Feb 2020 16:18:40 +0100
From: Michael Mraka <michael.mraka at redhat.com>
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo
Message-ID: <20200212151840.GD3775 at magni.redhat.com>
Content-Type: text/plain; charset=utf-8
Ree, Jan-Albert van:
> Good afternoon,
> While testing spacewalk-nightly we saw the same behaviour :
>
> - packages get downloaded just fine from remote repositories
> - building local repo data fails however
>
> Scenario's tested were both a clean install of spacewalk-nightly and an upgrade of 2.9 to nightly. With the upgrade from 2.9 to nightly, package sync was working fine before upgrading to nightly so it's definately something wrong in the nightly repo creation.
>
> Hope this helps, willing to test/debug further but need some pointers
> on what to test/report
Hello,
Most likely it's a bug https://bugzilla.redhat.com/show_bug.cgi?id=1802135.
I've fixed it today and updated backend packages are alredy in nightly repo.
Regards,
> --
> Jan-Albert van Ree
>
>
> ________________________________________
> From: spacewalk-list-bounces at redhat.com
> <spacewalk-list-bounces at redhat.com> on behalf of Jody McIvor
> <JMcIvor at bclc.com>
> Sent: Monday, January 13, 2020 17:22
> To: spacewalk-list at redhat.com
> Subject: [Spacewalk-list] Spacewalk 2.10 Failing to sync repo
>
> Hi Guys,
>
> Our DEV environment instance of spacewalk 2.10 seems to be failing to sync repositories. Running it via SSH reports success, and running it via HTTP interface reports success, yet logs show otherwise. As in the log snippet below, it is finding the packages, downloading them (seemingly) but failing to update the software channel(s).
...
> V- This issue seems to be affecting ALL software channels
> - Original repository location likely ruled out as the problem, as
> other repo's hosted on same machine, using same createrepo process are
> all working fine
> - Spacewalk shows proper "Last Modified" date, but "Last Repo Build"
> shows as way outdated. (Channels>Select Channel>DetailsTab)
> - Adding original repository location on the client (Effectively
> bypassing spacewalk) succeeds and allows access to all the latest RPMs
>
> Taskomatic log after attempting sync:
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,760 [Thread-53] INFO com.redhat.rhn.taskomatic.task.RepoSyncTask - 08:03:29 ======================================
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 | Channel: ? ? ? ?-channel-dev
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 ======================================
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Sync of channel started.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Processing repository with URL: http:// ? ? ? ?/rhel/x86_64/? ? ? ?-dev/
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages in repo: 88
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages already synced: 66
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Packages to sync: 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 New packages to download: 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 Downloading packages:
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 1/5 : ? ? ? ?-1.02-4.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 2/5 : ? ? ? ?-2.11.03.3.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 3/5 : ? ? ? ?-2.13.01.95.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 4/5 : ? ? ? ?-v2.13.03-1-1.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:29 5/5 : ? ? ? ?-v1.03.01-1-1.x86_64.rpm
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Importing packages to DB:
> INFO | jvm 1 | 2020/01/13 08:03:30 |
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages: |##########----------------------------------------| 20.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages: |####################------------------------------| 40.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages: |##############################--------------------| 60.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages: |########################################----------| 80.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | Importing packages: |##################################################| 100.0%
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Errata in repo: 0.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Sync of channel completed in 0:00:01.
> INFO | jvm 1 | 2020/01/13 08:03:30 | 08:03:30 Total time: 0:00:01
> INFO | jvm 1 | 2020/01/13 08:03:30 |
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Executing a task threw an exception: org.quartz.JobExecutionException
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Message: Command '[/usr/bin/spacewalk-repo-sync, --channel, ? ? ? ?-channel-dev, --type, yum]' exited with error code 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Cause: null
> INFO | jvm 1 | 2020/01/13 08:03:30 | 2020-01-13 08:03:30,763 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Stack trace:org.quartz.JobExecutionException: Command '[/usr/bin/spacewalk-repo-sync, --channel, ? ? ? ?-channel-dev, --type, yum]' exited with error code 5
> INFO | jvm 1 | 2020/01/13 08:03:30 | at com.redhat.rhn.taskomatic.task.RhnJavaJob.executeExtCmd(RhnJavaJob.java:103)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at com.redhat.rhn.taskomatic.task.RepoSyncTask.execute(RepoSyncTask.java:70)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
> INFO | jvm 1 | 2020/01/13 08:03:30 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
>
>
> Been poking at this one for a while now, any help appreciated!
>
> Thanks,
>
> Jody McIvor
> Sr. Systems Administrator, Integration Les Services D?int?gration
> [TEL] (250) 852-5202
>
> _______________________________________________
> Spacewalk-list mailing list
> Spacewalk-list at redhat.com
> https://www.redhat.com/mailman/listinfo/spacewalk-list
--
Michael Mr?ka
System Management Engineering, Red Hat
------------------------------
_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list
End of Spacewalk-list Digest, Vol 141, Issue 14
***********************************************
________________________________
This email is intended only for the addressee. It may contain confidential or proprietary information that cannot be disclosed without BCLC's permission. If you have received this email in error, please notify the sender immediately and delete the email.
More information about the Spacewalk-list
mailing list