<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> Looks like it was the OOM reaper killing the task. Thanks for pointing me in the right direction Brian! <pre class="moz-signature" cols="72"> David Gersting Linux Systems Administrator WVU Information Technology Services </pre> <div class="moz-cite-prefix">On 12/12/16 1:13 PM, Brian Bouterse wrote: </div> <blockquote cite="mid:CAAcvrTHuvSowXuRJ+xzJYNMtK9OT=xY-j2+EN3Rn2hshtSyz7A@mail.gmail.com" type="cite"> <div dir="ltr"> <div>I think the celery worker is experiencing a segfault or maybe it's being killed by the OOM. If the OOM is killing it there would be log evidence. If it's a segfault, with Python a segfault is unlikely, so this is probably a segfault while calling to the system using subprocess which Pulp does in various places. I haven't looked in the publish code of platform and rpm to look for subprocess usage but that would probably hint at the problem. To really debug something like that you would want capture a coredump. I think celery has the ability to capture coredumps, but I've never done it. </div> <div>The pulp-smash tests for publish showed they were working. Is it possible that this could be an environment issue? Is it possible to reproduce the issue on separate hardware to rule that out. If it is reproducable, I recommend opening a bug [0]. </div> <div> [0]: <a moz-do-not-send="true" href="https://pulp.plan.io/projects/pulp/issues/new" target="_blank">https://pulp.plan.io/projects/pulp/issues/new</a> </div> -Brian </div> <div class="gmail_extra"> <div class="gmail_quote">On Mon, Dec 12, 2016 at 11:49 AM, David Gersting <<a moz-do-not-send="true" href="mailto:dgersting@systems.wvu.edu" target="_blank">dgersting@systems.wvu.edu</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello everyone, I've been banging my head against the desk for a while on this one, and could use the group's help. I have a rather large repo (OEL 6's base repo with 36,684 RPMs) that I'm trying to mirror locally to speed up our os patching, and every time I try to publish the repo the task fails just after the "Publishing Delta RPMs" step starts. After some digging it seems to me that the worker is timing out. Has anyone else seen this and/or know how I can fix it or increase the timeout for this task? I've attached the full shell output for anyone who wants it, but the error message I'm seeing from the worker is: # journalctl --unit=pulp_worker-5 *SNIP* Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) Task pulp.server.managers.repo.publish.publish[e3d25854-757c-40af-8979-d0b7287263ed] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',) Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) Traceback (most recent call last): Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) human_status(exitcode)), Dec 12 10:48:19 *HOSTNAME* pulp[1403]: celery.worker.job:ERROR: (1403-27776) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL). Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. Se Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776) "MongoClient opened before fork. Create MongoClient " Dec 12 10:48:21 *HOSTNAME* pulp[49191]: py.warnings:WARNING: (49191-27776) Dec 12 10:48:22 *HOSTNAME* pulp[49191]: pulp.server.async.tasks:INFO: Task failed : [e3d25854-757c-40af-8979-d0b7287263ed] Any help would be much appreciated! -- David Gersting Linux Systems Administrator WVU Information Technology Services _______________________________________________ Pulp-list mailing list <a moz-do-not-send="true" href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </blockquote> </div> </div> </blockquote> </body> </html>