[Pulp-dev] Port Pulp3 to use RQ

Brian Bouterse bbouters at redhat.com
Wed May 16 15:22:19 UTC 2018 

With the switch to RQ I closed outstanding Pulp3-only, celery related
issues, 11 total, see below.  There are also 25 pulp2 celery issues that I
found. The pulp2 issues are still open and listed below also. I think it's
just a matter of time until those are closed for Pulp2 also, so if others
think it's right to close those issues please do.

FYI three nice RQ things. 1. 'rq info' is a great tool to see what the
state is of your Pulp tasking system.  2. RQ has a nice independant
scheduler for running reoccuring tasks https://github.com/rq/rq-scheduler.
3. You can have a dashboard with monitoring of your Pulp tasking system
with the https://github.com/rq/django-rq package.


# pulp2 celery-related open issues
https://pulp.plan.io/issues/1799
https://pulp.plan.io/issues/1679
https://pulp.plan.io/issues/1655
https://pulp.plan.io/issues/1400
https://pulp.plan.io/issues/1405
https://pulp.plan.io/issues/1370
https://pulp.plan.io/issues/1330
https://pulp.plan.io/issues/1202
https://pulp.plan.io/issues/1046
https://pulp.plan.io/issues/1020
https://pulp.plan.io/issues/857
https://pulp.plan.io/issues/766
https://pulp.plan.io/issues/678
https://pulp.plan.io/issues/1967
https://pulp.plan.io/issues/2664
https://pulp.plan.io/issues/2582
https://pulp.plan.io/issues/2492
https://pulp.plan.io/issues/2849
https://pulp.plan.io/issues/2837
https://pulp.plan.io/issues/2749
https://pulp.plan.io/issues/3048
https://pulp.plan.io/issues/2990
https://pulp.plan.io/issues/3540
https://pulp.plan.io/issues/3358
https://pulp.plan.io/issues/3281


# pulp3 issues
https://pulp.plan.io/issues/2988
https://pulp.plan.io/issues/3291
https://pulp.plan.io/issues/2259
https://pulp.plan.io/issues/3134
https://pulp.plan.io/issues/3292
https://pulp.plan.io/issues/3340
https://pulp.plan.io/issues/3656
https://pulp.plan.io/issues/2371
https://pulp.plan.io/issues/3361
https://pulp.plan.io/issues/2149
https://pulp.plan.io/issues/2632




On Mon, May 14, 2018 at 5:10 PM, David Davis <daviddavis at redhat.com> wrote:

> Great, thanks.
>
>
> David
>
> On Mon, May 14, 2018 at 4:46 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> If you are using the dev environment produced by the pulp/devel repo then
>> doing a vagrant destroy and vagrant up is the easiest way. Just make sure
>> you fetched all the latest changes from all the repos.
>>
>> If you want to port an existing environment into RQ, you should:
>>
>> 1. install RQ (the special commit version, see the current installation
>> docs)
>> 2. update the systemd files (see the examples in the docs)
>> 3. restart your workers
>>
>> If there are issues during ^ steps reach out and we can help resolve them.
>>
>> On Mon, May 14, 2018 at 4:39 PM, David Davis <daviddavis at redhat.com>
>> wrote:
>>
>>> Is there any way to convert an existing dev environment to use rq?
>>>
>>> Or do I just need to vagrant destroy and vagrant up?
>>>
>>>
>>> David
>>>
>>> On Mon, May 14, 2018 at 4:02 PM, Daniel Alley <dalley at redhat.com> wrote:
>>>
>>>> And pulp_python
>>>>
>>>> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>>
>>>>> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
>>>>> and merged pulp_ansible. This will be released with beta 3 of core coming
>>>>> out this Wednesday.
>>>>>
>>>>> If anyone runs into any issues please reach out via IRC or the mailing
>>>>> list.
>>>>>
>>>>> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse <bbouters at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> There's been a slight change in schedule. Now we believe the lowest
>>>>>> risk option is to merge today instead of tomorrow.
>>>>>>
>>>>>> We're finishing the latest rebase now, letting Travis tell us it's
>>>>>> good, and then merging it. We'll send a final note to the list post merge.
>>>>>>
>>>>>> Thanks to everyone for helping out!
>>>>>>
>>>>>> On Mon, May 14, 2018 at 12:07 PM, Dana Walker <dawalker at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>>>>>>> review/testing, and blog post.
>>>>>>>
>>>>>>> Dana Walker
>>>>>>>
>>>>>>> Associate Software Engineer
>>>>>>>
>>>>>>> Red Hat
>>>>>>>
>>>>>>> <https://www.redhat.com>
>>>>>>> <https://red.ht/sig>
>>>>>>>
>>>>>>> On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Great work on this. Also, thanks for announcing this on pulp-dev
>>>>>>>> well in advance.
>>>>>>>>
>>>>>>>>
>>>>>>>> David
>>>>>>>>
>>>>>>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> dalley has learned how to do some debugging already, so maybe he
>>>>>>>>> can
>>>>>>>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>>>>>>>> link to in a blog post - and  also point out the good demo
>>>>>>>>> @bmbouter
>>>>>>>>> put together for pulp 2.
>>>>>>>>>
>>>>>>>>> great job @dalley & @bmbouter on the blog post!
>>>>>>>>>
>>>>>>>>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>>>>>>>>> > At the proper time, a demo about the Pulp 3 task system will be
>>>>>>>>> very
>>>>>>>>> > beneficial. I am thinking about something similar what it was
>>>>>>>>> done for Pulp
>>>>>>>>> > 2.
>>>>>>>>> >
>>>>>>>>> > Looking forward for this.
>>>>>>>>> >
>>>>>>>>> > Regards,
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
>>>>>>>>> bbouters at redhat.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> All PRs have Travis showing green and all necessary LGTMs. The
>>>>>>>>> plan is to
>>>>>>>>> >> merge next Tuesday the 15th, which means it will be in core
>>>>>>>>> Beta 4.
>>>>>>>>> >>
>>>>>>>>> >> Yesterday, @dalley and I published a blog post which outlines
>>>>>>>>> the change
>>>>>>>>> >> for users along with a porting guide for plugins to port onto
>>>>>>>>> RQ as well.
>>>>>>>>> >>
>>>>>>>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>>>>>>>> >>
>>>>>>>>> >> Thank you to everyone for the help, collaboration, and energy
>>>>>>>>> on this
>>>>>>>>> >> significant change.
>>>>>>>>> >>
>>>>>>>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> I've finished my review and resolved all of the 'blocker'
>>>>>>>>> issues that
>>>>>>>>> >>> were uncovered during testing.  Overall, I'm highly confident
>>>>>>>>> that this is a
>>>>>>>>> >>> better path forwards than the continued use of Celery /
>>>>>>>>> Kombu.  There are a
>>>>>>>>> >>> couple of outstanding edge cases to be resolved eventually,
>>>>>>>>> which I plan to
>>>>>>>>> >>> file as issues post-merge, but nothing serious or intractable.
>>>>>>>>> >>>
>>>>>>>>> >>> If there are no objections, I think it would be reasonable to
>>>>>>>>> merge this
>>>>>>>>> >>> code after this week's beta builds are published (after, in
>>>>>>>>> order to avoid
>>>>>>>>> >>> major changes during Summit / PyCon prep time).
>>>>>>>>> >>>
>>>>>>>>> >>> Thank you, Brian, for doing the planning and work needed to
>>>>>>>>> make this
>>>>>>>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>>>>>>>> >>>
>>>>>>>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>>>>>>>> bbouters at redhat.com>
>>>>>>>>> >>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Through several rebases, now all PRs are showing the RQ PRs
>>>>>>>>> on Travis as
>>>>>>>>> >>>> passing with pulp-smash. Several points of feedback have been
>>>>>>>>> addressed.
>>>>>>>>> >>>>
>>>>>>>>> >>>> If you're interested in commenting on these PRs or trying
>>>>>>>>> them out,
>>>>>>>>> >>>> please do. I hope to merge after the other taking system
>>>>>>>>> maintainers @dalley
>>>>>>>>> >>>> and @daviddavis have finished their testing/review and
>>>>>>>>> barring any other
>>>>>>>>> >>>> calls for delay or blocking concerns.
>>>>>>>>> >>>>
>>>>>>>>> >>>> If there are any questions, issues, or concerns, please reach
>>>>>>>>> out, and
>>>>>>>>> >>>> we can talk through them.
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>>>>>>>> bbouters at redhat.com>
>>>>>>>>> >>>> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I put together a prototype and posted the PRs. I'm still
>>>>>>>>> working to get
>>>>>>>>> >>>>> Travis happy, but locally 100% of smash tests using these
>>>>>>>>> branches. It's
>>>>>>>>> >>>>> worked very reliably for me so far. There are no gaps in the
>>>>>>>>> pulp feature
>>>>>>>>> >>>>> set on top of RQ.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I hope people test it out and give some feedback. See the
>>>>>>>>> commit
>>>>>>>>> >>>>> messages for details on what was done. Here are the PRs:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> https://github.com/pulp/pulp/pull/3454
>>>>>>>>> >>>>> https://github.com/pulp/pulp_file/pull/72
>>>>>>>>> >>>>> https://github.com/pulp/devel/pull/146
>>>>>>>>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Feel free to send questions here or to the PR. Any feedback
>>>>>>>>> is welcome.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> -Brian
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>>>>>>>> mkovacik at redhat.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/
>>>>>>>>> esp.
>>>>>>>>> >>>>>> there's Fakeredis ;)
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> --
>>>>>>>>> >>>>>> milan
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>>>>>>>>> bbouters at redhat.com>
>>>>>>>>> >>>>>> wrote:
>>>>>>>>> >>>>>> > Thanks for all the discussion both on list and on irc.
>>>>>>>>> After more
>>>>>>>>> >>>>>> > investigation, it sounds like there are no feature gaps,
>>>>>>>>> but we will
>>>>>>>>> >>>>>> > need to
>>>>>>>>> >>>>>> > incorporate this workaround to cancel a task that is
>>>>>>>>> already
>>>>>>>>> >>>>>> > running.
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> > The feedback I've heard on the idea is that it's valuable
>>>>>>>>> and looks
>>>>>>>>> >>>>>> > feasible, but we won't really know until we prototype it
>>>>>>>>> a bit.
>>>>>>>>> >>>>>> > Based on the
>>>>>>>>> >>>>>> > technical outline in the previous email, I believe it can
>>>>>>>>> be
>>>>>>>>> >>>>>> > prototyped in a
>>>>>>>>> >>>>>> > day or two. I plan to do this soon, once I contribute to
>>>>>>>>> a few other
>>>>>>>>> >>>>>> > required-for-beta planning items first. I'll post my PR
>>>>>>>>> to see what
>>>>>>>>> >>>>>> > other
>>>>>>>>> >>>>>> > think of the change, probably next week.
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <
>>>>>>>>> dalley at redhat.com>
>>>>>>>>> >>>>>> > wrote:
>>>>>>>>> >>>>>> >>
>>>>>>>>> >>>>>> >> I meant in the sense that, what is the aftermath when it
>>>>>>>>> comes back
>>>>>>>>> >>>>>> >> online, and is it screwed up in ways that cause side
>>>>>>>>> effects.
>>>>>>>>> >>>>>> >>
>>>>>>>>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>>>>>>>>> jaudet at redhat.com>
>>>>>>>>> >>>>>> >> wrote:
>>>>>>>>> >>>>>> >>>
>>>>>>>>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the
>>>>>>>>> worker a
>>>>>>>>> >>>>>> >>> > SIGINT,
>>>>>>>>> >>>>>> >>> > it will finish the task and then stop processing new
>>>>>>>>> ones.  If
>>>>>>>>> >>>>>> >>> > you send the
>>>>>>>>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't
>>>>>>>>> think it
>>>>>>>>> >>>>>> >>> > gracefully
>>>>>>>>> >>>>>> >>> > handles this circumstance.
>>>>>>>>> >>>>>> >>>
>>>>>>>>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't
>>>>>>>>> catch that
>>>>>>>>> >>>>>> >>> signal.
>>>>>>>>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>>>>>>> >>>>>> >>>
>>>>>>>>> >>>>>> >>> If one is looking for a way to gracefully, immediately
>>>>>>>>> kill an RQ
>>>>>>>>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally,
>>>>>>>>> many processes
>>>>>>>>> >>>>>> >>> handle this signal in a hurried fashion. Semantically,
>>>>>>>>> this is
>>>>>>>>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal
>>>>>>>>> (Ctrl+c
>>>>>>>>> >>>>>> >>> sends
>>>>>>>>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>>>>>>> >>>>>> >>
>>>>>>>>> >>>>>> >>
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>> > _______________________________________________
>>>>>>>>> >>>>>> > Pulp-dev mailing list
>>>>>>>>> >>>>>> > Pulp-dev at redhat.com
>>>>>>>>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>> >>>>>> >
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> _______________________________________________
>>>>>>>>> >> Pulp-dev mailing list
>>>>>>>>> >> Pulp-dev at redhat.com
>>>>>>>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > Pulp-dev mailing list
>>>>>>>>> > Pulp-dev at redhat.com
>>>>>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pulp-dev mailing list
>>>>>>>>> Pulp-dev at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180516/6a3d939b/attachment.htm>

More information about the Pulp-dev mailing list