[Pulp-dev] Port Pulp3 to use RQ

Wed May 9 15:29:54 UTC 2018

dalley has learned how to do some debugging already, so maybe he can
look at doing a demo. Good suggestion, Kersom. It would be good to
link to in a blog post - and  also point out the good demo @bmbouter
put together for pulp 2.

great job @dalley & @bmbouter on the blog post!

On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
> At the proper time, a demo about the Pulp 3 task system will be very
> beneficial. I am thinking about something similar what it was done for Pulp
> 2.
>
> Looking forward for this.
>
> Regards,
>
>
>
>
>
>
> On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <bbouters at redhat.com> wrote:
>>
>> All PRs have Travis showing green and all necessary LGTMs. The plan is to
>> merge next Tuesday the 15th, which means it will be in core Beta 4.
>>
>> Yesterday, @dalley and I published a blog post which outlines the change
>> for users along with a porting guide for plugins to port onto RQ as well.
>>
>> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>
>> Thank you to everyone for the help, collaboration, and energy on this
>> significant change.
>>
>> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com> wrote:
>>>
>>> I've finished my review and resolved all of the 'blocker' issues that
>>> were uncovered during testing.  Overall, I'm highly confident that this is a
>>> better path forwards than the continued use of Celery / Kombu.  There are a
>>> couple of outstanding edge cases to be resolved eventually, which I plan to
>>> file as issues post-merge, but nothing serious or intractable.
>>>
>>> If there are no objections, I think it would be reasonable to merge this
>>> code after this week's beta builds are published (after, in order to avoid
>>> major changes during Summit / PyCon prep time).
>>>
>>> Thank you, Brian, for doing the planning and work needed to make this
>>> happen.  It was a lot of effort and is very highly appreciated.
>>>
>>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com>
>>> wrote:
>>>>
>>>> Through several rebases, now all PRs are showing the RQ PRs on Travis as
>>>> passing with pulp-smash. Several points of feedback have been addressed.
>>>>
>>>> If you're interested in commenting on these PRs or trying them out,
>>>> please do. I hope to merge after the other taking system maintainers @dalley
>>>> and @daviddavis have finished their testing/review and barring any other
>>>> calls for delay or blocking concerns.
>>>>
>>>> If there are any questions, issues, or concerns, please reach out, and
>>>> we can talk through them.
>>>>
>>>>
>>>>
>>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>>>
>>>>> I put together a prototype and posted the PRs. I'm still working to get
>>>>> Travis happy, but locally 100% of smash tests using these branches. It's
>>>>> worked very reliably for me so far. There are no gaps in the pulp feature
>>>>> set on top of RQ.
>>>>>
>>>>> I hope people test it out and give some feedback. See the commit
>>>>> messages for details on what was done. Here are the PRs:
>>>>>
>>>>> https://github.com/pulp/pulp/pull/3454
>>>>> https://github.com/pulp/pulp_file/pull/72
>>>>> https://github.com/pulp/devel/pull/146
>>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>>>
>>>>> Feel free to send questions here or to the PR. Any feedback is welcome.
>>>>>
>>>>> -Brian
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <mkovacik at redhat.com>
>>>>> wrote:
>>>>>>
>>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>>>>> there's Fakeredis ;)
>>>>>>
>>>>>>
>>>>>> --
>>>>>> milan
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <bbouters at redhat.com>
>>>>>> wrote:
>>>>>> > Thanks for all the discussion both on list and on irc. After more
>>>>>> > investigation, it sounds like there are no feature gaps, but we will
>>>>>> > need to
>>>>>> > incorporate this workaround to cancel a task that is already
>>>>>> > running.
>>>>>> >
>>>>>> > The feedback I've heard on the idea is that it's valuable and looks
>>>>>> > feasible, but we won't really know until we prototype it a bit.
>>>>>> > Based on the
>>>>>> > technical outline in the previous email, I believe it can be
>>>>>> > prototyped in a
>>>>>> > day or two. I plan to do this soon, once I contribute to a few other
>>>>>> > required-for-beta planning items first. I'll post my PR to see what
>>>>>> > other
>>>>>> > think of the change, probably next week.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <dalley at redhat.com>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >> I meant in the sense that, what is the aftermath when it comes back
>>>>>> >> online, and is it screwed up in ways that cause side effects.
>>>>>> >>
>>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <jaudet at redhat.com>
>>>>>> >> wrote:
>>>>>> >>>
>>>>>> >>> > RQ does not support revoking tasks.  If you send the worker a
>>>>>> >>> > SIGINT,
>>>>>> >>> > it will finish the task and then stop processing new ones.  If
>>>>>> >>> > you send the
>>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't think it
>>>>>> >>> > gracefully
>>>>>> >>> > handles this circumstance.
>>>>>> >>>
>>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch that
>>>>>> >>> signal.
>>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>>>> >>>
>>>>>> >>> If one is looking for a way to gracefully, immediately kill an RQ
>>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many processes
>>>>>> >>> handle this signal in a hurried fashion. Semantically, this is
>>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c
>>>>>> >>> sends
>>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>>>> >>
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Pulp-dev mailing list
>>>>>> > Pulp-dev at redhat.com
>>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>