[Pulp-list] Pulp publish task timeout?
David Gersting
dgersting at systems.wvu.edu
Mon Dec 12 16:49:54 UTC 2016
Hello everyone,
I've been banging my head against the desk for a while on this one, and
could use the group's help.
I have a rather large repo (OEL 6's base repo with 36,684 RPMs) that I'm
trying to mirror locally to speed up our os patching, and every time I
try to publish the repo the task fails just after the "Publishing Delta
RPMs" step starts. After some digging it seems to me that the worker is
timing out. Has anyone else seen this and/or know how I can fix it or
increase the timeout for this task?
I've attached the full shell output for anyone who wants it, but the
error message I'm seeing from the worker is:
# journalctl --unit=pulp_worker-5
*SNIP*
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR:
(1403-27776) Task
pulp.server.managers.repo.publish.publish[e3d25854-757c-40af-8979-d0b7287263ed]
raised unexpected: WorkerLostError('Worker exited prematurely: signal 9
(SIGKILL).',)
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR:
(1403-27776) Traceback (most recent call last):
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR:
(1403-27776) File
"/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in
mark_as_worker_lost
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR:
(1403-27776) human_status(exitcode)),
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR:
(1403-27776) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING:
(49191-27776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74:
UserWarning: MongoClient opened before fork. Create MongoClient with
connect=False, or create client after forking. Se
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING:
(49191-27776) "MongoClient opened before fork. Create MongoClient "
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING:
(49191-27776)
Dec 12 10:48:22 *HOSTNAME* pulp[49191]:
pulp.server.async.tasks:INFO: Task failed :
[e3d25854-757c-40af-8979-d0b7287263ed]
Any help would be much appreciated!
--
David Gersting
Linux Systems Administrator
WVU Information Technology Services
-------------- next part --------------
# pulp-admin rpm repo publish run --repo-id=mirror-oracle-ol6-x86_64-base
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Publishing Repository [mirror-oracle-ol6-x86_64-base]
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[\]
... completed
Publishing RPMs
[===========================================================================================================================================] 100%
36684 of 36684 items
Task Failed
The task status e3d25854-757c-40af-8979-d0b7287263ed exited immediately for some reason. Marking as errored. Check the logs for more details
# pulp-admin tasks details --task-id=e3d25854-757c-40af-8979-d0b7287263ed
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Operations: publish
Resources: mirror-oracle-ol6-x86_64-base (repository)
State: Failed
Start Time: 2016-12-12T14:40:03Z
Finish Time: 2016-12-12T15:48:22Z
Result: N/A
Task Id: e3d25854-757c-40af-8979-d0b7287263ed
Progress Report:
Yum Distributor:
Description: Initializing repo metadata
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 1
Num Success: 1
State: FINISHED
Step Id: 4c40c7c9-ae7f-4a7e-ba83-5d14701b88e3
Step Type: initialize_repo_metadata
Description: Publishing Distribution files
Details:
Error Details:
Items Total: 0
Num Failures: 0
Num Processed: 0
Num Success: 0
State: FINISHED
Step Id: b01e0d95-650c-469d-aa0d-a17dda4d871b
Step Type: distribution
Description: Publishing RPMs
Details:
Error Details:
Items Total: 36684
Num Failures: 0
Num Processed: 36684
Num Success: 36684
State: IN_PROGRESS
Step Id: b18d0587-9e34-47c5-9b37-c2920fda9c35
Step Type: rpms
Description: Publishing Delta RPMs
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 9507bc46-42d7-4523-9aff-65621490caf7
Step Type: drpms
Description: Publishing Errata
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 06062abb-087d-4d64-9f21-1942dff3053d
Step Type: errata
Description: Publishing Comps file
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: ad9b425e-55b9-408e-8a04-0f77b86afe0b
Step Type: comps
Description: Publishing Metadata.
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 033dcf01-fc62-4002-bb6d-a4b09d0dfe06
Step Type: metadata
Description: Closing repo metadata
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: b5fd28af-6058-41b7-8d3c-7f586dec143f
Step Type: close_repo_metadata
Description: Generating sqlite files
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 9663bfbf-3479-4dd7-a081-634002b16daa
Step Type: generate sqlite
Description: Generating HTML files
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 132e93fe-8fc4-4112-99b7-c1ed1f0cd0d1
Step Type: repoview
Description: Publishing files to web
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: b529e375-2b88-4716-9672-6ad847afa4da
Step Type: publish_directory
Description: Writing Listings File
Details:
Error Details:
Items Total: 1
Num Failures: 0
Num Processed: 0
Num Success: 0
State: NOT_STARTED
Step Id: 84aa847c-ba40-4fe2-85bb-113bd6131e5c
Step Type: initialize_repo_metadata
# journalctl --unit=pulp_worker-5
*snip*
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) Task pulp.server.managers.repo.publish.publish[e3d25854-757c-40af-8979-d0b7287263ed] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) Traceback (most recent call last):
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) human_status(exitcode)),
Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. Se
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776) "MongoClient opened before fork. Create MongoClient "
Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776)
Dec 12 10:48:22 *HOSTNAME* pulp[49191]: pulp.server.async.tasks:INFO: Task failed : [e3d25854-757c-40af-8979-d0b7287263ed]
More information about the Pulp-list
mailing list