<div dir="ltr"><div><div>Probably my recommendation is not the best, but it works :D </div>When my sync run stays "stuck" I apply <a href="https://pulp.plan.io/issues/686">https://pulp.plan.io/issues/686</a> which is: $ cat all.sh sudo service qpidd $1 sudo service pulp_celerybeat $1 sudo service pulp_resource_manager $1 sudo service pulp_workers $1 sudo service httpd $1 1. $ ./all.sh restart </div>Then if there are tasks waiting I cancel them and relaunch. If you are using CentOS7 I recommend to open a second terminal and launch journalctl -xf to see pulp behavior </div><div class="gmail_extra"> <div class="gmail_quote">On Sat, Apr 9, 2016 at 5:33 AM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">After you reboot and restart all Pulp services I expect there to be 0 messages on the 'resource_manager' queue. Can you show the queue depths of your broker in those situations for all queues? With Qpid you can do this with `qpid-stat -q` if I remember correctly. RabbitMQ has a similar command but I don't know it. Also for those "stuck" tasks that are not starting when you expect them too, can you see if they are "assigned" to a worker. This doesn't show up in pulp-admin output[0] of a task detail so instead use a command like this to show the task details by uuid. pulp-admin -vv tasks details --task-id a83b32c4-4cb1-439e-a373-91797d0b185a The -vv part shows the actual webserver response which will contain a line like: "worker_name": "reserved_resource_worker-0@dev", This info would be helpful in resolving your issue. [0]: <a href="https://pulp.plan.io/issues/1832" rel="noreferrer" target="_blank">https://pulp.plan.io/issues/1832</a> -Brian On 04/08/2016 03:15 PM, Matthew Madey wrote: > When I checked on the state, the status was "not started". I think > that's why it remained stuck after reboots and bouncing the services. > > On Fri, Apr 8, 2016 at 1:27 PM, Brian Bouterse <<a href="mailto:bbouters@redhat.com">bbouters@redhat.com</a> <div><div class="h5">> <mailto:<a href="mailto:bbouters@redhat.com">bbouters@redhat.com</a>>> wrote: > > I'm not sure how you would get into this situations. When it occurs can > you check which worker is assigned the work, and verify that that worker > is still running? > > Upon starting a worker will move previous tasks it was handling that are > still in the running state to cancelled. Also pulp_celerybeat monitors > pulp workers to determine if died to move its tasks to a cancelled > state. Both of these mechanisms would have to fail in order to have a > task stay in the running state when its not running. Could it still be > running? > > You indicate you rebooted the box and those tasks didn't go to > cancelled. Is it possible they are on another box connected to your > broker or when killing them it didn't respond to the signal you sent it? > > The 2.7.1 doesn't have any known defects like the ones your describing > so sending more info to the list would be good. > > -Brian > > > On 03/30/2016 05:58 PM, Matthew Madey wrote: > > I have a job that mistakenly thinks it's still running.. > > > > # pulp-admin -u admin -p ************ rpm repo sync run > > --repo-id=rhel-x86_64-server-7-base-tools > > > +----------------------------------------------------------------------+ > > Synchronizing Repository [rhel-x86_64-server-7-base-tools] > > > +----------------------------------------------------------------------+ > > > > A sync task is already in progress for this repository. Its > progress will be > > tracked below. > > > > This command may be exited via ctrl+c without affecting the request. > > > > [/] > > Waiting to begin... > > > > > > I checked all running processes and there is no repo sync currently > > running. I have even gone so far as to delete the repo and > recreate it.. > > same issue. I have also tried running pulp-admin orphan remove --all, > > which executes successfully, but does not fix the problem. I have also > > tried rebooting the server, bouncing all pulp services.. still no joy. > > I'm guessing there is a file somewhere that tracks pending tasks? How > > can I clear this so I can successfully run the job again? I'm running > > Pulp 2.7.1-1 > > > > > > _______________________________________________ > > Pulp-list mailing list </div></div>> > <a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a> <mailto:<a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a>> > > <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> > > > > _______________________________________________ > Pulp-list mailing list > <a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a> <mailto:<a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a>> <div class="HOEnZb"><div class="h5">> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> > > _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </div></div></blockquote></div> </div>