[Pulp-list] Pulp sync failure - not well-formed (invalid token): line 44436, column 0

[Paul Jochum]

Hi All:

     For the last 2 weeks, one of my daily pulp sync's has been 
failing.  It is always the same repo (9 other syncs on that server run 
fine).  All of the sync's are to Red Hat, for different RHEL 7 
repositories.  The one that is failing is for rhel-x86_64-server-7-source.

This is what I see for the failing sync:
*****************************************************************************

$ pulp-admin -u admin -p admin rpm repo sync run --repo-id=$i
+----------------------------------------------------------------------+
          Synchronizing Repository [rhel-x86_64-server-7-source]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[\]
... failed

not well-formed (invalid token): line 44436, column 0


Task Failed

Importer indicated a failed response

*****************************************************************************
And this is in /var/log/messages:
Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received 
task: 
pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5]
Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.job:INFO: Task 
pulp.server.async.tasks._reserve_resource[94539898-05e5-4800-aa61-b75aa05a38c5] 
succeeded in 0.0402820348972s: 
u'reserved_resource_worker-1 at lss-pulp02.ih.lucent.com.dq'
Mar 23 07:43:39 lss-pulp02 pulp: celery.worker.strategy:INFO: Received 
task: 
pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997]
Mar 23 07:43:39 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from 
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/source/SRPMS/.
Mar 23 07:43:39 lss-pulp02 pulp: 
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS 
connection (1): cdn.redhat.com
Mar 23 07:43:40 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Mar 23 07:43:40 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Mar 23 07:43:40 lss-pulp02 pulp: 
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS 
connection (1): cdn.redhat.com
Mar 23 07:43:40 lss-pulp02 pulp: 
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS 
connection (1): cdn.redhat.com
Mar 23 07:43:40 lss-pulp02 pulp: 
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS 
connection (1): cdn.redhat.com
Mar 23 07:43:40 lss-pulp02 pulp: 
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS 
connection (1): cdn.redhat.com
Mar 23 07:43:42 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call 
last):
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", 
line 105, in run
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:     metadata_files = 
self.get_metadata()
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", 
line 212, in get_metadata
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files.generate_dbs()
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py", 
line 275, in generate_dbs
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:     for element in generator:
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", 
line 53, in package_list_generator
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:     for event, element in 
xml_iterator:
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR:   File "<string>", line 64, 
in __iter__
Mar 23 07:43:45 lss-pulp02 pulp: 
pulp_rpm.plugins.importers.yum.sync:ERROR: SyntaxError: not well-formed 
(invalid token): line 44436, column 0
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Task 
pulp.server.tasks.repository.sync_with_auto_publish[e8fe2c7a-ee33-4548-bbf4-176e2018a997] 
raised unexpected: PulpExecutionException('Importer indicated a failed 
response',)
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: Traceback 
(most recent call last):
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File 
"/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in 
trace_task
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     R = retval 
= fun(*args, **kwargs)
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 316, 
in __call__
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     return 
super(Task, self).__call__(*args, **kwargs)
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File 
"/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in 
__protected_call__
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     return 
self.run(*args, **kwargs)
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line 
210, in sync_with_auto_publish
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: sync_result = 
managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides)
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:   File 
"/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", 
line 113, in sync
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR:     raise 
PulpExecutionException(_('Importer indicated a failed response'))
Mar 23 07:43:45 lss-pulp02 pulp: celery.worker.job:ERROR: 
PulpExecutionException: Importer indicated a failed response

*****************************************************************************

Any suggestions on how to clean it up?  Would any additional information 
help?

thanks,
Paul