[Pulp-dev] Port Pulp3 to use RQ

Mon May 14 19:55:44 UTC 2018

RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported and
merged pulp_ansible. This will be released with beta 3 of core coming out
this Wednesday.

If anyone runs into any issues please reach out via IRC or the mailing list.

On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse <bbouters at redhat.com>
wrote:

> There's been a slight change in schedule. Now we believe the lowest risk
> option is to merge today instead of tomorrow.
>
> We're finishing the latest rebase now, letting Travis tell us it's good,
> and then merging it. We'll send a final note to the list post merge.
>
> Thanks to everyone for helping out!
>
> On Mon, May 14, 2018 at 12:07 PM, Dana Walker <dawalker at redhat.com> wrote:
>
>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>> review/testing, and blog post.
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> <https://www.redhat.com>
>> <https://red.ht/sig>
>>
>> On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com>
>> wrote:
>>
>>> Great work on this. Also, thanks for announcing this on pulp-dev well in
>>> advance.
>>>
>>>
>>> David
>>>
>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:
>>>
>>>> dalley has learned how to do some debugging already, so maybe he can
>>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>>> link to in a blog post - and  also point out the good demo @bmbouter
>>>> put together for pulp 2.
>>>>
>>>> great job @dalley & @bmbouter on the blog post!
>>>>
>>>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>>>> > At the proper time, a demo about the Pulp 3 task system will be very
>>>> > beneficial. I am thinking about something similar what it was done
>>>> for Pulp
>>>> > 2.
>>>> >
>>>> > Looking forward for this.
>>>> >
>>>> > Regards,
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>> >>
>>>> >> All PRs have Travis showing green and all necessary LGTMs. The plan
>>>> is to
>>>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>>>> >>
>>>> >> Yesterday, @dalley and I published a blog post which outlines the
>>>> change
>>>> >> for users along with a porting guide for plugins to port onto RQ as
>>>> well.
>>>> >>
>>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>>> >>
>>>> >> Thank you to everyone for the help, collaboration, and energy on this
>>>> >> significant change.
>>>> >>
>>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>>>> wrote:
>>>> >>>
>>>> >>> I've finished my review and resolved all of the 'blocker' issues
>>>> that
>>>> >>> were uncovered during testing.  Overall, I'm highly confident that
>>>> this is a
>>>> >>> better path forwards than the continued use of Celery / Kombu.
>>>> There are a
>>>> >>> couple of outstanding edge cases to be resolved eventually, which I
>>>> plan to
>>>> >>> file as issues post-merge, but nothing serious or intractable.
>>>> >>>
>>>> >>> If there are no objections, I think it would be reasonable to merge
>>>> this
>>>> >>> code after this week's beta builds are published (after, in order
>>>> to avoid
>>>> >>> major changes during Summit / PyCon prep time).
>>>> >>>
>>>> >>> Thank you, Brian, for doing the planning and work needed to make
>>>> this
>>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>>> >>>
>>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>>> bbouters at redhat.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Through several rebases, now all PRs are showing the RQ PRs on
>>>> Travis as
>>>> >>>> passing with pulp-smash. Several points of feedback have been
>>>> addressed.
>>>> >>>>
>>>> >>>> If you're interested in commenting on these PRs or trying them out,
>>>> >>>> please do. I hope to merge after the other taking system
>>>> maintainers @dalley
>>>> >>>> and @daviddavis have finished their testing/review and barring any
>>>> other
>>>> >>>> calls for delay or blocking concerns.
>>>> >>>>
>>>> >>>> If there are any questions, issues, or concerns, please reach out,
>>>> and
>>>> >>>> we can talk through them.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>>> bbouters at redhat.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> I put together a prototype and posted the PRs. I'm still working
>>>> to get
>>>> >>>>> Travis happy, but locally 100% of smash tests using these
>>>> branches. It's
>>>> >>>>> worked very reliably for me so far. There are no gaps in the pulp
>>>> feature
>>>> >>>>> set on top of RQ.
>>>> >>>>>
>>>> >>>>> I hope people test it out and give some feedback. See the commit
>>>> >>>>> messages for details on what was done. Here are the PRs:
>>>> >>>>>
>>>> >>>>> https://github.com/pulp/pulp/pull/3454
>>>> >>>>> https://github.com/pulp/pulp_file/pull/72
>>>> >>>>> https://github.com/pulp/devel/pull/146
>>>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>> >>>>>
>>>> >>>>> Feel free to send questions here or to the PR. Any feedback is
>>>> welcome.
>>>> >>>>>
>>>> >>>>> -Brian
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>>> mkovacik at redhat.com>
>>>> >>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>>> >>>>>> there's Fakeredis ;)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> milan
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>>>> bbouters at redhat.com>
>>>> >>>>>> wrote:
>>>> >>>>>> > Thanks for all the discussion both on list and on irc. After
>>>> more
>>>> >>>>>> > investigation, it sounds like there are no feature gaps, but
>>>> we will
>>>> >>>>>> > need to
>>>> >>>>>> > incorporate this workaround to cancel a task that is already
>>>> >>>>>> > running.
>>>> >>>>>> >
>>>> >>>>>> > The feedback I've heard on the idea is that it's valuable and
>>>> looks
>>>> >>>>>> > feasible, but we won't really know until we prototype it a bit.
>>>> >>>>>> > Based on the
>>>> >>>>>> > technical outline in the previous email, I believe it can be
>>>> >>>>>> > prototyped in a
>>>> >>>>>> > day or two. I plan to do this soon, once I contribute to a few
>>>> other
>>>> >>>>>> > required-for-beta planning items first. I'll post my PR to see
>>>> what
>>>> >>>>>> > other
>>>> >>>>>> > think of the change, probably next week.
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <
>>>> dalley at redhat.com>
>>>> >>>>>> > wrote:
>>>> >>>>>> >>
>>>> >>>>>> >> I meant in the sense that, what is the aftermath when it
>>>> comes back
>>>> >>>>>> >> online, and is it screwed up in ways that cause side effects.
>>>> >>>>>> >>
>>>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>>>> jaudet at redhat.com>
>>>> >>>>>> >> wrote:
>>>> >>>>>> >>>
>>>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the
>>>> worker a
>>>> >>>>>> >>> > SIGINT,
>>>> >>>>>> >>> > it will finish the task and then stop processing new
>>>> ones.  If
>>>> >>>>>> >>> > you send the
>>>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't
>>>> think it
>>>> >>>>>> >>> > gracefully
>>>> >>>>>> >>> > handles this circumstance.
>>>> >>>>>> >>>
>>>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch
>>>> that
>>>> >>>>>> >>> signal.
>>>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>> >>>>>> >>>
>>>> >>>>>> >>> If one is looking for a way to gracefully, immediately kill
>>>> an RQ
>>>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
>>>> processes
>>>> >>>>>> >>> handle this signal in a hurried fashion. Semantically, this
>>>> is
>>>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal
>>>> (Ctrl+c
>>>> >>>>>> >>> sends
>>>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>> >>>>>> >>
>>>> >>>>>> >>
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> > _______________________________________________
>>>> >>>>>> > Pulp-dev mailing list
>>>> >>>>>> > Pulp-dev at redhat.com
>>>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>> >>>>>> >
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> Pulp-dev mailing list
>>>> >> Pulp-dev at redhat.com
>>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>> >>
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Pulp-dev mailing list
>>>> > Pulp-dev at redhat.com
>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>> >
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180514/632c4f77/attachment.htm>