[Pulp-list] pulp 2.8 repos went into waiting state and never ends
Mallick, Samiron
samiron.mallick at gmail.com
Wed May 4 04:07:34 UTC 2016
Hey Brian, thanks for the reply.
*From the below output I could see "resource_worker-1" is responsible for
this task and I have 4 worker displayed on the server.*
# pulp-admin tasks list
+----------------------------------------------------------------------+
Tasks
+----------------------------------------------------------------------+
Operations: sync
Resources: rhel-6-server-supplementary-rpms (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697
# pulp-admin -vv tasks details --task-id
49b83f70-e6d6-4cdb-9c5a-93c20c31d697
+----------------------------------------------------------------------+
Task Details
+----------------------------------------------------------------------+
2016-05-04 04:55:33,231 - DEBUG - sending GET request to
/pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/
2016-05-04 04:55:33,362 - INFO - GET request to
/pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/ with parameters
None
2016-05-04 04:55:33,362 - INFO - Response status : 200
2016-05-04 04:55:33,363 - INFO - Response body :
{
"exception": null,
"task_type": "pulp.server.managers.repo.sync.sync",
"_href": "/pulp/api/v2/tasks/49b83f70-e6d6-4cdb-9c5a-93c20c31d697/",
"task_id": "49b83f70-e6d6-4cdb-9c5a-93c20c31d697",
"tags": [
"pulp:repository:rhel-6-server-supplementary-rpms",
"pulp:action:sync"
],
"finish_time": null,
"_ns": "task_status",
"start_time": null,
"traceback": null,
"spawned_tasks": [],
"progress_report": {},
"queue": "reserved_resource_worker-1 at mysrv.dq",
"state": "waiting",
"worker_name": "reserved_resource_worker-1 at mysrv",
"result": null,
"error": null,
"_id": {
"$oid": "572964399b70a2ea1d2694aa"
},
"id": "572964399b70a2ea1d2694aa"
}
Operations: sync
Resources: rhel-6-server-supplementary-rpms (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Result: Incomplete
Task Id: 49b83f70-e6d6-4cdb-9c5a-93c20c31d697
Progress Report:
# pulp-admin status
+----------------------------------------------------------------------+
Status of the server
+----------------------------------------------------------------------+
Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:53:34Z
_id: reserved_resource_worker-3 at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:54:00Z
_id: reserved_resource_worker-2 at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:54:00Z
_id: resource_manager at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:54:00Z
_id: reserved_resource_worker-1 at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:54:01Z
_id: reserved_resource_worker-0 at mysrv
_ns: workers
Last Heartbeat: 2016-05-04T02:54:03Z
Messaging Connection:
Connected: True
Versions:
Platform Version: 2.8.2
# ps -awfux | grep celery
root 4637 0.0 0.0 112644 960 pts/0 S+ 04:56
0:00 \_ grep --color=auto celery
apache 1592 0.0 1.4 667716 56368 ? Ssl May03 0:26
/usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n
resource_manager@%h -Q resource_manager -c 1 --events --umask 18
--pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
apache 2921 0.0 1.4 667664 54296 ? Sl May03 0:13 \_
/usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n
resource_manager@%h -Q resource_manager -c 1 --events --umask 18
--pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
apache 1616 0.0 1.4 667996 56400 ? Ssl May03 0:27
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-0.pid
--heartbeat-interval=30
apache 2919 0.0 1.4 741536 54564 ? Sl May03 0:11 \_
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-0.pid
--heartbeat-interval=30
apache 1626 0.0 1.5 668560 59524 ? Ssl May03 0:29
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-1.pid
--heartbeat-interval=30
apache 4561 0.0 1.4 668560 56260 ? S 04:47 0:00 \_
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-1.pid
--heartbeat-interval=30
apache 1631 0.0 1.5 667748 58508 ? Ssl May03 0:27
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-2.pid
--heartbeat-interval=30
apache 2922 4.2 8.0 1042956 311476 ? Sl May03 48:25 \_
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-2.pid
--heartbeat-interval=30
apache 1637 0.0 1.4 667744 56368 ? Ssl May03 0:27
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-3.pid
--heartbeat-interval=30
apache 2920 0.0 1.4 815420 54760 ? Sl May03 0:13 \_
/usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A
pulp.server.async.app -c 1 --events --umask 18
--pidfile=/var/run/pulp/reserved_resource_worker-3.pid
--heartbeat-interval=30
apache 4620 6.5 0.8 663652 31432 ? Ssl 04:56 0:00
/usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
*As I saw errors in output of pulp_worker-1, I restarted each worker
individually, and all error seems gone.*
# systemctl status pulp_workers.service
● pulp_workers.service - Pulp Celery Workers
Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled;
vendor preset: disabled)
Active: active (exited) since Wed 2016-05-04 05:36:38 CEST; 3s ago
Process: 5717 ExecStop=/usr/bin/python -m
pulp.server.async.manage_workers stop (code=exited, status=0/SUCCESS)
Process: 5731 ExecStart=/usr/bin/python -m
pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)
Main PID: 5731 (code=exited, status=0/SUCCESS)
May 04 05:36:38 mysrv systemd[1]: Starting Pulp Celery Workers...
May 04 05:36:38 mysrv systemd[1]: Started Pulp Celery Workers.
# systemctl status pulp_worker-0
? pulp_worker-0.service - Pulp Worker #0
Loaded: loaded (/run/systemd/system/pulp_worker-0.service; static;
vendor preset: disabled)
Active: active (running) since Wed 2016-05-04 05:10:44 CEST; 1min 26s ago
Main PID: 4753 (celery)
CGroup: /system.slice/pulp_worker-0.service
+-4753 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
+-4766 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> transport:
qpid://mysrv:5672//
May 04 05:10:46 mysrv celery[4753]: - ** ---------- .> results: disabled
May 04 05:10:46 mysrv celery[4753]: - *** --- * --- .> concurrency: 1
(prefork)
May 04 05:10:46 mysrv celery[4753]: -- ******* ----
May 04 05:10:46 mysrv celery[4753]: --- ***** ----- [queues]
May 04 05:10:46 mysrv celery[4753]: -------------- .> celery
exchange=celery(direct) key=celery
May 04 05:10:46 mysrv celery[4753]: .> reserved_resource_worker-0 at mysrv.dq
exchange=C.dq(direct) key=rese...s <http://sim.biz/>rv
May 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
May 04 05:10:46 mysrv pulp[4753]: celery.worker.consumer:INFO: Connected to
qpid://mysrv:5672//
May 04 05:10:46 mysrv pulp[4753]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
Hint: Some lines were ellipsized, use -l to show in full.
# systemctl status pulp_worker-1
? pulp_worker-1.service - Pulp Worker #1
Loaded: loaded (/run/systemd/system/pulp_worker-1.service; static;
vendor preset: disabled)
Active: active (running) since Wed 2016-05-04 05:08:16 CEST; 3min 57s ago
Main PID: 4718 (celery)
CGroup: /system.slice/pulp_worker-1.service
+-4718 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
+-4733 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...3cc3c36]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...ce7430b]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.job:INFO: Task
pulp.server.controllers.repository.download_deferred[aad88f32-...9s: None
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.db.reaper.reap_expired_documents[02...8322faa]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...ddadf87]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...d0cf8c6]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...72edf98]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...1e9e4bc]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.download_def...30f8627]
May 04 05:08:23 mysrv pulp[4718]: celery.worker.strategy:INFO: Received
task: pulp.server.controllers.repository.queue_downlo...fd23e13]
Hint: Some lines were ellipsized, use -l to show in full.
# systemctl status pulp_worker-2
? pulp_worker-2.service - Pulp Worker #2
Loaded: loaded (/run/systemd/system/pulp_worker-2.service; static;
vendor preset: disabled)
Active: active (running) since Wed 2016-05-04 05:11:06 CEST; 1min 10s ago
Main PID: 4776 (celery)
CGroup: /system.slice/pulp_worker-2.service
+-4776 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
+-4789 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> transport:
qpid://mysrv:5672//
May 04 05:11:07 mysrv celery[4776]: - ** ---------- .> results: disabled
May 04 05:11:07 mysrv celery[4776]: - *** --- * --- .> concurrency: 1
(prefork)
May 04 05:11:07 mysrv celery[4776]: -- ******* ----
May 04 05:11:07 mysrv celery[4776]: --- ***** ----- [queues]
May 04 05:11:07 mysrv celery[4776]: -------------- .> celery
exchange=celery(direct) key=celery
May 04 05:11:07 mysrv celery[4776]: .>
reserved_resource_worker-2@<redacted by list administrator>
exchange=C.dq(direct) key=rese...s <http://sim.biz/>rv
May 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
May 04 05:11:07 mysrv pulp[4776]: celery.worker.consumer:INFO: Connected to
qpid://mysrv:5672//
May 04 05:11:07 mysrv pulp[4776]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
Hint: Some lines were ellipsized, use -l to show in full.
# systemctl status pulp_worker-3
? pulp_worker-3.service - Pulp Worker #3
Loaded: loaded (/run/systemd/system/pulp_worker-3.service; static;
vendor preset: disabled)
Active: active (running) since Wed 2016-05-04 05:11:21 CEST; 59s ago
Main PID: 4798 (celery)
CGroup: /system.slice/pulp_worker-3.service
+-4798 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
+-4811 /usr/bin/python /usr/bin/celery worker -n
reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events
--umask 18 --pidfile=/var...
May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> transport:
qpid://mysrv:5672//
May 04 05:11:22 mysrv celery[4798]: - ** ---------- .> results: disabled
May 04 05:11:22 mysrv celery[4798]: - *** --- * --- .> concurrency: 1
(prefork)
May 04 05:11:22 mysrv celery[4798]: -- ******* ----
May 04 05:11:22 mysrv celery[4798]: --- ***** ----- [queues]
May 04 05:11:22 mysrv celery[4798]: -------------- .> celery
exchange=celery(direct) key=celery
May 04 05:11:22 mysrv celery[4798]: .>
reserved_resource_worker-3@<redacted by list administrator>
exchange=C.dq(direct) key=rese...s <http://sim.biz/>rv
May 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
May 04 05:11:22 mysrv pulp[4798]: celery.worker.consumer:INFO: Connected to
qpid://mysrv:5672//
May 04 05:11:22 mysrv pulp[4798]: kombu.transport.qpid:INFO: Connected to
qpid with SASL mechanism ANONYMOUS
Hint: Some lines were ellipsized, use -l to show in full.
*Now I have no tasks queued at all. I ran the repo sync again. and again it
went to waiting.....*
# pulp-admin tasks list
+----------------------------------------------------------------------+
Tasks
+----------------------------------------------------------------------+
No tasks found
# pulp-admin rpm repo sync run --repo-id=rhel-6-server-supplementary-rpms
+----------------------------------------------------------------------+
Synchronizing Repository [rhel-6-server-supplementary-rpms]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
[/]
Waiting to begin...
On Wed, May 4, 2016 at 1:52 AM, Brian Bouterse <bbouters at redhat.com> wrote:
> Kodiak is right that the second task stuck at "Waiting to Begin" is
> likely waiting behind another operation on that same repo. Canceling the
> one prior will likely allow the later one to start.
>
> How many workers are running and how many do you expect? You can see
> what Pulp thinks with: `pulp-admin status`
>
> You can compare that to your pulp processes on all of your Pulp servers
> with `sudo ps -awfux | grep celery`.
>
> Also you can look at the task details with -vv to see the worker the
> halted task is assigned to. Something like `pulp-admin -vv tasks details
> --task-id 03842c9d-e053-4a6f-a4c4-2d7302be9c8c.`
>
> Unfortunately you'll have to see the worker in the raw response with -vv
> because of [0].
>
> [0]: https://pulp.plan.io/issues/1832
>
> -Brian
>
>
> On 05/03/2016 11:53 AM, Kodiak Firesmith wrote:
> > I believe you may need to cancel the pending repo sync task before you
> > can delete the repo. Maybe try:
> > pulp-admin tasks cancel --task-id=2d776d63-fd8a-4e0a-8f32-d2276c85187c
> > pulp-admin tasks cancel --task-id=03842c9d-e053-4a6f-a4c4-2d7302be9c8c
> >
> > Then:
> > pulp-admin rpm repo delete --repo-id=rhel-6-server-supplementary-rpms
> >
> >
> > On Tue, May 3, 2016 at 11:47 AM, Mallick, Samiron
> > <samiron.mallick at gmail.com <mailto:samiron.mallick at gmail.com>> wrote:
> >
> > Could anyone please tell me what went wrong with the repository. One
> > of my EL7 server registered and was able to fetch contents from CDN.
> > Recently I found one of the repo stuck after downloading RPMs. It’s
> > never ending. I rebooted my server, cancelled tasks, deleted the
> > repo and recreated, but no luck. No if I run sync, it directly going
> > to waiting stage. Earlier I observed it was starting the task but
> > the start time was same as I ran the job first time. Even now I am
> > not able to delete the repo as well as it is showing “Waiting to
> > begin”. I am running Pulp v2.8. Any idea would be greatly
> appreciated.
> >
> >
> >
> > # rpm -qa pulp-server
> >
> > pulp-server-2.8.2-1.el7.noarch
> >
> >
> >
> > # pulp-admin rpm repo sync run
> > --repo-id=rhel-6-server-supplementary-rpms
> >
> >
> +----------------------------------------------------------------------+
> >
> > Synchronizing Repository [rhel-6-server-supplementary-rpms]
> >
> >
> +----------------------------------------------------------------------+
> >
> >
> >
> > This command may be exited via ctrl+c without affecting the request.
> >
> >
> >
> >
> >
> > Downloading metadata...
> >
> > [\]
> >
> > ... completed
> >
> >
> >
> > Downloading repository content...
> >
> > [-]
> >
> > [==================================================] 100%
> >
> > RPMs: 0/0 items
> >
> > Delta RPMs: 0/0 items
> >
> >
> >
> > ... completed
> >
> >
> >
> > Downloading distribution files...
> >
> > [==================================================] 100%
> >
> > Distributions: 0/0 items
> >
> > ... completed
> >
> >
> >
> > Importing errata...
> >
> > [/]
> >
> > ... completed
> >
> >
> >
> > Importing package groups/categories...
> >
> > [-]
> >
> > ... completed
> >
> >
> >
> > Cleaning duplicate packages...
> >
> > [|]
> >
> > ... completed
> >
> >
> >
> > *** AND STUCK HERE ***
> >
> >
> >
> > # pulp-admin tasks list
> >
> >
> +----------------------------------------------------------------------+
> >
> > Tasks
> >
> >
> +----------------------------------------------------------------------+
> >
> >
> >
> > Operations: sync
> >
> > Resources: rhel-6-server-supplementary-rpms (repository)
> >
> > State: Running
> >
> > Start Time: 2016-05-03T07:06:36Z
> >
> > Finish Time: Incomplete
> >
> > Task Id: 2d776d63-fd8a-4e0a-8f32-d2276c85187c
> >
> >
> >
> > Operations: publish
> >
> > Resources: rhel-6-server-supplementary-rpms (repository)
> >
> > State: Waiting
> >
> > Start Time: Unstarted
> >
> > Finish Time: Incomplete
> >
> > Task Id: 03842c9d-e053-4a6f-a4c4-2d7302be9c8c
> >
> >
> >
> > # date
> >
> > Tue May 3 09:22:30 CEST 2016
> >
> > # pulp-admin rpm repo sync schedules list
> > --repo-id=rhel-6-server-supplementary-rpms
> >
> >
> +----------------------------------------------------------------------+
> >
> > Schedules
> >
> >
> +----------------------------------------------------------------------+
> >
> >
> >
> > There are no schedules defined for this operation.
> >
> >
> >
> > # pulp-admin rpm repo delete
> --repo-id=rhel-6-server-supplementary-rpms
> >
> > This command may be exited via ctrl+c without affecting the request.
> >
> >
> >
> >
> >
> > [-]
> >
> > Running...
> >
> > [-]
> >
> > Waiting to begin...
> >
> >
> >
> > *** AND STUCK HERE ***
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
> >
> >
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> >
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20160504/b218626b/attachment.htm>
More information about the Pulp-list
mailing list