[Pulp-list] 2.8 Tasks Stuck at Waiting

[Brian Bouterse]

To anyone who has a stuck task running Pulp 2.6 or newer, can you open a
bug or e-mail the list with the following. The full output from:

pulp-admin -vv tasks details --task-id <the_stuck_task_UUID>

Post the full output, but observe the value of "worker_name" similar to
what was suggested here[0].

Can you also include the `sudo ps -awfux | grep celery` output to show
your running process?

Also can you grep the logs for that UUID and include any lines that you
see as output too.

[0]: https://www.redhat.com/archives/pulp-list/2016-April/msg00026.html

Thanks,
Brian


On 04/11/2016 01:32 PM, Mihai Ibanescu wrote:
> I have a similar problem on 2.7.1 with rabbitmq.
> 
> Occasionally a task will get "lost":
> 
> pulp-admin tasks details --task-id 622041ac-e9e4-4a15-bd7c-7c98a17782e0
> +----------------------------------------------------------------------+
>                               Task Details
> +----------------------------------------------------------------------+
> 
> Operations:       publish
> Resources:        thirdparty-snapshot-rpm-latest (repository)
> State:            Waiting
> Start Time:       Unstarted
> Finish Time:      Incomplete
> Result:           Incomplete
> Task Id:          622041ac-e9e4-4a15-bd7c-7c98a17782e0
> Progress Report:  
> 
> 
> I have seen more than one getting in this state.
> 
> I can cancel the task, but I'd like to find out if I can tickle
> something to re-process the queue. Other tasks do get correctly processed.
> 
> If there's something you would like me to do in order to debug this
> further, please let me know.
> 
> Thank you!
> Mihai
> 
> On Fri, Apr 8, 2016 at 9:27 AM, Brian Bouterse <bbouters at redhat.com
> <mailto:bbouters at redhat.com>> wrote:
> 
>     There is a known issue [0] introduced in 2.8.0 which will cause your
>     tasks to become stuck at waiting. This occurs after Pulp runs normally
>     for hours or days.
> 
>     A fix is completed but won't be released until Pulp 2.8.3 is released.
>     The good news is, you can resolve that issue now by upgrading
>     python-kombu on your existing 2.8.0, 2.8.1, or 2.8.2 system. Manually
>     install python-kombu-3.0.33-5.pulp for your distro on your system,
>     restart Pulp services and you'll be all set. All of the links below come
>     out of the Pulp build system[1]
> 
>     EL6: http://tinyurl.com/j5sy5zh
>     EL7: http://tinyurl.com/jppwcdv
>     F22: http://tinyurl.com/hp7z4sq
>     F23: http://tinyurl.com/zplzua3
>     F24: update is in -testing repo [2]
>     rawhide: already available in stable
> 
>     These are the same versions of python-kombu that you'll receive when
>     2.8.3 comes out so upgrading this package will keep your packages on the
>     normal upgrade path. These newer python-kombu versions are compatible
>     with 2.8.0, 2.8.1, and 2.8.2 versions of Pulp.
> 
>     [0]: https://pulp.plan.io/issues/1801
>     [1]: http://koji.katello.org/koji/packageinfo?packageID=665
>     [2]: https://bodhi.fedoraproject.org/updates/FEDORA-2016-ec038bbf19
> 
>     -Brian
> 
>     _______________________________________________
>     Pulp-list mailing list
>     Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-list
> 
>