<div dir="ltr">Hey Brian, thanks for the reply. <div> </div><div>From the below output I could see "resource_worker-1" is responsible for this task and I have 4 worker displayed on the server. # pulp-admin tasks list+----------------------------------------------------------------------+ Tasks+----------------------------------------------------------------------+ Operations: syncResources: rhel-6-server-supplementary-rpms (repository)State: WaitingStart Time: UnstartedFinish Time: IncompleteTask Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697 # pulp-admin -vv tasks details --task-id 49b83f70-e6d6-4cdb-9c5a-93c20c31d697+----------------------------------------------------------------------+ Task Details+----------------------------------------------------------------------+ 2016-05-04 04:55:33,231 - DEBUG - sending GET request to /pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/2016-05-04 04:55:33,362 - INFO - GET request to /pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/ with parameters None2016-05-04 04:55:33,362 - INFO - Response status : 200 2016-05-04 04:55:33,363 - INFO - Response body :{ "exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/", "task_id": "49b83f70-e6d6-4cdb-9c5a-93c20c31d697", "tags": [ "pulp:repository:rhel-6-server-supplementary-rpms", "pulp:action:sync" ], "finish_time": null, "_ns": "task_status", "start_time": null, "traceback": null, "spawned_tasks": [], "progress_report": {}, "queue": "reserved_resource_worker-1@mysrv.dq", "state": "waiting", "worker_name": "reserved_resource_worker-1@mysrv", "result": null, "error": null, "_id": { "$oid": "572964399b70a2ea1d2694aa" }, "id": "572964399b70a2ea1d2694aa"} Operations: syncResources: rhel-6-server-supplementary-rpms (repository)State: WaitingStart Time: UnstartedFinish Time: IncompleteResult: IncompleteTask Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697Progress Report: # pulp-admin status+----------------------------------------------------------------------+ Status of the server+----------------------------------------------------------------------+ Api Version: 2Database Connection: Connected: TrueKnown Workers: _id: scheduler@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:53:34Z _id: reserved_resource_worker-3@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:54:00Z _id: reserved_resource_worker-2@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:54:00Z _id: resource_manager@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:54:00Z _id: reserved_resource_worker-1@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:54:01Z _id: reserved_resource_worker-0@mysrv _ns: workers Last Heartbeat: 2016-05-04T02:54:03ZMessaging Connection: Connected: TrueVersions: Platform Version: 2.8.2 # ps -awfux | grep celeryroot 4637 0.0 0.0 112644 960 pts/0 S+ 04:56 0:00 \_ grep --color=auto celeryapache 1592 0.0 1.4 667716 56368 ? Ssl May03 0:26 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30apache 2921 0.0 1.4 667664 54296 ? Sl May03 0:13 \_ /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30apache 1616 0.0 1.4 667996 56400 ? Ssl May03 0:27 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30apache 2919 0.0 1.4 741536 54564 ? Sl May03 0:11 \_ /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30apache 1626 0.0 1.5 668560 59524 ? Ssl May03 0:29 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid --heartbeat-interval=30apache 4561 0.0 1.4 668560 56260 ? S 04:47 0:00 \_ /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid --heartbeat-interval=30apache 1631 0.0 1.5 667748 58508 ? Ssl May03 0:27 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid --heartbeat-interval=30apache 2922 4.2 8.0 1042956 311476 ? Sl May03 48:25 \_ /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid --heartbeat-interval=30apache 1637 0.0 1.4 667744 56368 ? Ssl May03 0:27 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid --heartbeat-interval=30apache 2920 0.0 1.4 815420 54760 ? Sl May03 0:13 \_ /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid --heartbeat-interval=30apache 4620 6.5 0.8 663652 31432 ? Ssl 04:56 0:00 /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler As I saw errors in output of pulp_worker-1, I restarted each worker individually, and all error seems gone. # systemctl status pulp_workers.service● pulp_workers.service - Pulp Celery Workers Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2016-05-04 05:36:38 CEST; 3s ago Process: 5717 ExecStop=/usr/bin/python -m pulp.server.async.manage_workers stop (code=exited, status=0/SUCCESS) Process: 5731 ExecStart=/usr/bin/python -m pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)Main PID: 5731 (code=exited, status=0/SUCCESS) May 04 05:36:38 mysrv systemd[1]: Starting Pulp Celery Workers...May 04 05:36:38 mysrv systemd[1]: Started Pulp Celery Workers. # systemctl status pulp_worker-0? pulp_worker-0.service - Pulp Worker #0 Loaded: loaded (/run/systemd/system/pulp_worker-0.service; static; vendor preset: disabled) Active: active (running) since Wed 2016-05-04 05:10:44 CEST; 1min 26s agoMain PID: 4753 (celery) CGroup: /system.slice/pulp_worker-0.service +-4753 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... +-4766 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> transport: qpid://mysrv<a href="" target="_blank">:5672//</a>May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> results: disabledMay 04 05:10:46 mysrv celery[4753]: - *** --- * --- .> concurrency: 1 (prefork)May 04 05:10:46 mysrv celery[4753]: -- ******* ----May 04 05:10:46 mysrv celery[4753]: --- ***** ----- [queues]May 04 05:10:46 mysrv celery[4753]: -------------- .> celery exchange=celery(direct) key=celeryMay 04 05:10:46 mysrv celery[4753]: .> reserved_resource_worker-0@mysrv.dq exchange=C.dq(direct) key=rese...<a href="http://sim.biz/" target="_blank">s</a>rvMay 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSMay 04 05:10:46 mysrv pulp[4753]: celery.worker.consumer:INFO: Connected to qpid://mysrv<a href="" target="_blank">:5672//</a>May 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSHint: Some lines were ellipsized, use -l to show in full. # systemctl status pulp_worker-1? pulp_worker-1.service - Pulp Worker #1 Loaded: loaded (/run/systemd/system/pulp_worker-1.service; static; vendor preset: disabled) Active: active (running) since Wed 2016-05-04 05:08:16 CEST; 3min 57s agoMain PID: 4718 (celery) CGroup: /system.slice/pulp_worker-1.service +-4718 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... +-4733 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...3cc3c36]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...ce7430b]May 04 05:08:23 mysrv pulp[4718]: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[aad88f32-...9s: NoneMay 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[02...8322faa]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...ddadf87]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...d0cf8c6]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...72edf98]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...1e9e4bc]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_def...30f8627]May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_downlo...fd23e13]Hint: Some lines were ellipsized, use -l to show in full. # systemctl status pulp_worker-2? pulp_worker-2.service - Pulp Worker #2 Loaded: loaded (/run/systemd/system/pulp_worker-2.service; static; vendor preset: disabled) Active: active (running) since Wed 2016-05-04 05:11:06 CEST; 1min 10s agoMain PID: 4776 (celery) CGroup: /system.slice/pulp_worker-2.service +-4776 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... +-4789 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> transport: qpid://mysrv<a href=":5672//" target="_blank">:5672//</a>May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> results: disabledMay 04 05:11:07 mysrv celery[4776]: - *** --- * --- .> concurrency: 1 (prefork)May 04 05:11:07 mysrv celery[4776]: -- ******* ----May 04 05:11:07 mysrv celery[4776]: --- ***** ----- [queues]May 04 05:11:07 mysrv celery[4776]: -------------- .> celery exchange=celery(direct) key=celeryMay 04 05:11:07 mysrv celery[4776]: .> reserved_resource_worker-2@srpadi exchange=C.dq(direct) key=rese...<a href="http://sim.biz/" target="_blank">s</a>rvMay 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSMay 04 05:11:07 mysrv pulp[4776]: celery.worker.consumer:INFO: Connected to qpid://mysrv<a href="" target="_blank">:5672//</a>May 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSHint: Some lines were ellipsized, use -l to show in full. # systemctl status pulp_worker-3? pulp_worker-3.service - Pulp Worker #3 Loaded: loaded (/run/systemd/system/pulp_worker-3.service; static; vendor preset: disabled) Active: active (running) since Wed 2016-05-04 05:11:21 CEST; 59s agoMain PID: 4798 (celery) CGroup: /system.slice/pulp_worker-3.service +-4798 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... +-4811 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var... May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> transport: qpid://mysrv<a href="" target="_blank">:5672//</a>May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> results: disabledMay 04 05:11:22 mysrv celery[4798]: - *** --- * --- .> concurrency: 1 (prefork)May 04 05:11:22 mysrv celery[4798]: -- ******* ----May 04 05:11:22 mysrv celery[4798]: --- ***** ----- [queues]May 04 05:11:22 mysrv celery[4798]: -------------- .> celery exchange=celery(direct) key=celeryMay 04 05:11:22 mysrv celery[4798]: .> reserved_resource_worker-3@<redacted by list administrator> exchange=C.dq(direct) key=rese...<a href="http://sim.biz/" target="_blank">s</a>rvMay 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSMay 04 05:11:22 mysrv pulp[4798]: celery.worker.consumer:INFO: Connected to qpid://mysrv<a href="" target="_blank">:5672//</a>May 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUSHint: Some lines were ellipsized, use -l to show in full. Now I have no tasks queued at all. I ran the repo sync again. and again it went to waiting..... # pulp-admin tasks list+----------------------------------------------------------------------+ Tasks+----------------------------------------------------------------------+ No tasks found # pulp-admin rpm repo sync run --repo-id=rhel-6-server-supplementary-rpms+----------------------------------------------------------------------+ Synchronizing Repository [rhel-6-server-supplementary-rpms]+----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. [/]Waiting to begin...</div><div class="gmail_extra"> <div><div><div dir="ltr"> </div></div></div> <div class="gmail_quote">On Wed, May 4, 2016 at 1:52 AM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Kodiak is right that the second task stuck at "Waiting to Begin" is likely waiting behind another operation on that same repo. Canceling the one prior will likely allow the later one to start. How many workers are running and how many do you expect? You can see what Pulp thinks with: `pulp-admin status` You can compare that to your pulp processes on all of your Pulp servers with `sudo ps -awfux | grep celery`. Also you can look at the task details with -vv to see the worker the halted task is assigned to. Something like `pulp-admin -vv tasks details --task-id 03842c9d-e053-4a6f-a4c4-2d7302be9c8c.` Unfortunately you'll have to see the worker in the raw response with -vv because of [0]. [0]: <a href="https://pulp.plan.io/issues/1832" rel="noreferrer" target="_blank">https://pulp.plan.io/issues/1832</a> -Brian On 05/03/2016 11:53 AM, Kodiak Firesmith wrote: > I believe you may need to cancel the pending repo sync task before you > can delete the repo. Maybe try: > pulp-admin tasks cancel --task-id=2d776d63-fd8a-4e0a-8f32-d2276c85187c > pulp-admin tasks cancel --task-id=03842c9d-e053-4a6f-a4c4-2d7302be9c8c > > Then: > pulp-admin rpm repo delete --repo-id=rhel-6-server-supplementary-rpms > > > On Tue, May 3, 2016 at 11:47 AM, Mallick, Samiron <div><div>> <<a href="mailto:samiron.mallick@gmail.com" target="_blank">samiron.mallick@gmail.com</a> <mailto:<a href="mailto:samiron.mallick@gmail.com" target="_blank">samiron.mallick@gmail.com</a>>> wrote: > > Could anyone please tell me what went wrong with the repository. One > of my EL7 server registered and was able to fetch contents from CDN. > Recently I found one of the repo stuck after downloading RPMs. It’s > never ending. I rebooted my server, cancelled tasks, deleted the > repo and recreated, but no luck. No if I run sync, it directly going > to waiting stage. Earlier I observed it was starting the task but > the start time was same as I ran the job first time. Even now I am > not able to delete the repo as well as it is showing “Waiting to > begin”. I am running Pulp v2.8. Any idea would be greatly appreciated. > > > > # rpm -qa pulp-server > > pulp-server-2.8.2-1.el7.noarch > > > > # pulp-admin rpm repo sync run > --repo-id=rhel-6-server-supplementary-rpms > > +----------------------------------------------------------------------+ > > Synchronizing Repository [rhel-6-server-supplementary-rpms] > > +----------------------------------------------------------------------+ > > > > This command may be exited via ctrl+c without affecting the request. > > > > > > Downloading metadata... > > [\] > > ... completed > > > > Downloading repository content... > > [-] > > [==================================================] 100% > > RPMs: 0/0 items > > Delta RPMs: 0/0 items > > > > ... completed > > > > Downloading distribution files... > > [==================================================] 100% > > Distributions: 0/0 items > > ... completed > > > > Importing errata... > > [/] > > ... completed > > > > Importing package groups/categories... > > [-] > > ... completed > > > > Cleaning duplicate packages... > > [|] > > ... completed > > > > *** AND STUCK HERE *** > > > > # pulp-admin tasks list > > +----------------------------------------------------------------------+ > > Tasks > > +----------------------------------------------------------------------+ > > > > Operations: sync > > Resources: rhel-6-server-supplementary-rpms (repository) > > State: Running > > Start Time: 2016-05-03T07:06:36Z > > Finish Time: Incomplete > > Task Id: 2d776d63-fd8a-4e0a-8f32-d2276c85187c > > > > Operations: publish > > Resources: rhel-6-server-supplementary-rpms (repository) > > State: Waiting > > Start Time: Unstarted > > Finish Time: Incomplete > > Task Id: 03842c9d-e053-4a6f-a4c4-2d7302be9c8c > > > > # date > > Tue May 3 09:22:30 CEST 2016 > > # pulp-admin rpm repo sync schedules list > --repo-id=rhel-6-server-supplementary-rpms > > +----------------------------------------------------------------------+ > > Schedules > > +----------------------------------------------------------------------+ > > > > There are no schedules defined for this operation. > > > > # pulp-admin rpm repo delete --repo-id=rhel-6-server-supplementary-rpms > > This command may be exited via ctrl+c without affecting the request. > > > > > > [-] > > Running... > > [-] > > Waiting to begin... > > > > *** AND STUCK HERE *** > > > _______________________________________________ > Pulp-list mailing list </div></div>> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> <mailto:<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a>> > <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> <div><div>> > > > > _______________________________________________ > Pulp-list mailing list > <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> > _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </div></div></blockquote></div> </div></div>