I’ve been having failures syncing with certain RH repositories as well. For me, it’s the 7Workstation and 5Client repo’s (7Server doesn’t have any problems). I get a “Forbidden”, which would seem to indicate a problem with the cert… but this sync problem is intermittent. If I manually kick off a sync, sometimes it works, often it doesn’t.
Mar 23 10:24:07 yummy pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[f8759f66-7416-4fd1-9ee4-6005ca06df8a] succeeded in 0.0339540460263s: None
Mar 23 10:24:07 yummy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/.
Mar 23 10:24:07 yummy pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) sync failed
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) Traceback (most recent call last):
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 104, in run
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) metadata_files = self.get_metadata()
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 198, in get_metadata
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) raise FailedException(str(e))
Mar 23 10:24:08 yummy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (1986-41632) FailedException: Forbidden
Mar 23 10:24:08 yummy pulp: celery.worker.job:ERROR: (1595-41632) Task pulp.server.managers.repo.sync.sync[80a6866e-0ed6-456a-8b7a-8d0cadbd84aa] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Here’s two back-to-back sync’s, first failed (messages above), second was fine.
# pulp-admin rpm repo sync run --repo-id=base-7Workstation
+----------------------------------------------------------------------+
Synchronizing Repository [base-7Workstation]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[-]
... failed
Forbidden
Task Failed
Importer indicated a failed response
# pulp-admin rpm repo sync run --repo-id=base-7Workstation
+----------------------------------------------------------------------+
Synchronizing Repository [base-7Workstation]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[/]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Copying files
[\]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[-]
... completed
Publishing Comps file
[==================================================] 100%
86 of 86 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
[|]
... completed
Publishing files to web
[|]
... completed
Writing Listings File
[-]
... completed
Writing Listings File
[\]
... completed
Task Succeeded
—-
The other issue I have is that I’m syncing the kickstart trees…. When I first setup Pulp, it was syncing 7.1 server and workstation. Early last week, my workstation repo looks to have fallen back to 7.0. Looking at the local workstation treeinfo file, it lists version=7.0, whereas my server treeinfo lists 7.1.
I used curl to see what’s on the RH repo, treeinfo for both workstation and server list 7.0. (I disabled my server sync so that it remains at 7.1)
These are the two feeds I’m syncing against:
https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/kickstart/
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/kickstart/
It’s weird, if I look at the redhat-release package, it’s 7.1:
# ls -al /var/lib/pulp/published/yum/http/repos/base-7Workstation/redhat-release-workstation-7.1-1.el7.x86_64.rpm
But the images/pxeboot files are all from 7.0
# ls -al /var/lib/pulp/content/distribution/ks-Red\ Hat\ Enterprise\ Linux-Workstation-7.0-x86_64/images/pxeboot/
total 72456
drwxr-xr-x. 2 apache apache 55 Mar 14 14:44 .
drwxr-xr-x. 3 apache apache 20 Mar 14 14:44 ..
-rw-r--r--. 1 apache apache 35544564 Mar 14 14:44 initrd.img
-rw-r--r--. 1 apache apache 33744152 Mar 14 14:44 upgrade.img
-rw-r--r--. 1 apache apache 4902000 Mar 14 14:44 vmlinuz
There is no longer a 7.1 workstation distribution directory.
# ls -al /var/lib/pulp/content/distribution/
total 8
drwxr-xr-x. 5 apache apache 4096 Mar 18 08:46 .
drwxr-xr-x. 6 apache apache 74 Mar 14 14:36 ..
drwxrwxr-x. 5 apache apache 4096 Mar 16 14:31 ks-Red Hat Enterprise Linux Client-5.11-x86_64
drwxrwxr-x. 5 apache apache 60 Mar 14 22:30 ks-Red Hat Enterprise Linux-Server-7.1-x86_64
drwxrwxr-x. 4 apache apache 47 Mar 14 14:44 ks-Red Hat Enterprise Linux-Workstation-7.0-x86_64
I’ve only been using Pulp for 2 weeks, so I’m learning as I go. The above doesn’t seem right, but maybe I don’t have something configured correctly?
Thanks for any help,
Reece
On 3/23/15, 7:39 AM, "Paul Jochum" <paul.jochum at alcatel-lucent.com> wrote:
>Hi Barnaby:
>
>$ rpm -qa | grep pulp
>python-pulp-rpm-common-2.5.3-1.el6.noarch
>pulp-server-2.5.3-1.el6.noarch
>pulp-puppet-plugins-2.5.3-1.el6.noarch
>pulp-selinux-2.5.3-1.el6.noarch
>python-isodate-0.5.0-4.pulp.el6.noarch
>python-kombu-3.0.15-12.pulp.el6.noarch
>python-pulp-puppet-common-2.5.3-1.el6.noarch
>python-pulp-bindings-2.5.3-1.el6.noarch
>pulp-admin-client-2.5.3-1.el6.noarch
>mod_wsgi-3.4-2.pulp.el6.x86_64
>pulp-rpm-plugins-2.5.3-1.el6.noarch
>pulp-puppet-admin-extensions-2.5.3-1.el6.noarch
>python-pulp-common-2.5.3-1.el6.noarch
>python-pulp-client-lib-2.5.3-1.el6.noarch
>pulp-rpm-admin-extensions-2.5.3-1.el6.noarch
>
>thanks,
>Paul
>
>
>On 03/23/2015 09:25 AM, Barnaby Court wrote:
>> Hi Paul, What version of Pulp are you running? "rpm -qa | grep pulp " should give all the pertinent information. Thanks!
>>
>> -Barnaby
>>
>> ----- Original Message -----
>> From: "Paul Jochum" <paul.jochum at alcatel-lucent.com>
>> To: pulp-list at redhat.com
>> Sent: Monday, March 23, 2015 8:50:48 AM
>> Subject: [Pulp-list] Pulp sync failure - not well-formed (invalid token): line 44436, column 0
>>
>> Hi All:
>>
>> For the last 2 weeks, one of my daily pulp sync's has been
>> failing. It is always the same repo (9 other syncs on that server run
>> fine). All of the sync's are to Red Hat, for different RHEL 7
>> repositories. The one that is failing is for rhel-x86_64-server-7-source.
>>
>> This is what I see for the failing sync:
>> *****************************************************************************
>>
>> $ pulp-admin -u admin -p admin rpm repo sync run --repo-id=$i
>> +----------------------------------------------------------------------+
>> Synchronizing Repository [rhel-x86_64-server-7-source]
>> +----------------------------------------------------------------------+
>>
>> This command may be exited via ctrl+c without affecting the request.
>>
>>
>> Downloading metadata...
>> [\]
>> ... failed
>>
>> not well-formed (invalid token): line 44436, column 0
>>
>>
>> Task Failed
>>
>> Importer indicated a failed response
>>
>> *****************************************************************************
>> And this is in /var/log/messages:
>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received
>> task:
>> pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5]
>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.job:INFO: Task
>> pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5]
>> succeeded in 0.0402820348972s:
>> u'reserved_resource_worker-1 at lss-pulp02.ih.lucent.com.dq'
>> Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received
>> task:
>> pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997]
>> Mar 23 07:43:39 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/source/SRPMS/.
>> Mar 23 07:43:39 lss-pulp02 pulp:
>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>> connection (1): cdn.redhat.com
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>> connection (1): cdn.redhat.com
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>> connection (1): cdn.redhat.com
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>> connection (1): cdn.redhat.com
>> Mar 23 07:43:40 lss-pulp02 pulp:
>> requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
>> connection (1): cdn.redhat.com
>> Mar 23 07:43:42 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call
>> last):
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
>> line 105, in run
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files =
>> self.get_metadata()
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
>> line 212, in get_metadata
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files.generate_dbs()
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py",
>> line 275, in generate_dbs
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: for element in generator:
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py",
>> line 53, in package_list_generator
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: for event, element in
>> xml_iterator:
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: File "<string>", line 64,
>> in __iter__
>> Mar 23 07:43:45 lss-pulp02 pulp:
>> pulp_rpm.plugins.importers.yum.sync:ERROR: SyntaxError: not well-formed
>> (invalid token): line 44436, column 0
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Task
>> pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997]
>> raised unexpected: PulpExecutionException('Importer indicated a failed
>> response',)
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Traceback
>> (most recent call last):
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: File
>> "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in
>> trace_task
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: R = retval
>> = fun(*args, **kwargs)
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 316,
>> in __call__
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: return
>> super(Task, self).__call__(*args, **kwargs)
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: File
>> "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in
>> __protected_call__
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: return
>> self.run(*args, **kwargs)
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line
>> 210, in sync_with_auto_publish
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: sync_result =
>> managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides)
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: File
>> "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py",
>> line 113, in sync
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: raise
>> PulpExecutionException(_('Importer indicated a failed response'))
>> Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:
>> PulpExecutionException: Importer indicated a failed response
>>
>> *****************************************************************************
>>
>> Any suggestions on how to clean it up? Would any additional information
>> help?
>>
>> thanks,
>> Paul
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>_______________________________________________
>Pulp-list mailing list
>Pulp-list at redhat.com
>https://www.redhat.com/mailman/listinfo/pulp-list