[Pulp-dev] Port Pulp3 to use RQ

Mon May 14 16:07:36 UTC 2018

+1 to advance notice, and +1 to @bmbouter and @dalley on the work,
review/testing, and blog post.

Dana Walker

Associate Software Engineer

Red Hat

<https://www.redhat.com>
<https://red.ht/sig>

On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com> wrote:

> Great work on this. Also, thanks for announcing this on pulp-dev well in
> advance.
>
>
> David
>
> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:
>
>> dalley has learned how to do some debugging already, so maybe he can
>> look at doing a demo. Good suggestion, Kersom. It would be good to
>> link to in a blog post - and  also point out the good demo @bmbouter
>> put together for pulp 2.
>>
>> great job @dalley & @bmbouter on the blog post!
>>
>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>> > At the proper time, a demo about the Pulp 3 task system will be very
>> > beneficial. I am thinking about something similar what it was done for
>> Pulp
>> > 2.
>> >
>> > Looking forward for this.
>> >
>> > Regards,
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>> >>
>> >> All PRs have Travis showing green and all necessary LGTMs. The plan is
>> to
>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>> >>
>> >> Yesterday, @dalley and I published a blog post which outlines the
>> change
>> >> for users along with a porting guide for plugins to port onto RQ as
>> well.
>> >>
>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>> >>
>> >> Thank you to everyone for the help, collaboration, and energy on this
>> >> significant change.
>> >>
>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>> wrote:
>> >>>
>> >>> I've finished my review and resolved all of the 'blocker' issues that
>> >>> were uncovered during testing.  Overall, I'm highly confident that
>> this is a
>> >>> better path forwards than the continued use of Celery / Kombu.  There
>> are a
>> >>> couple of outstanding edge cases to be resolved eventually, which I
>> plan to
>> >>> file as issues post-merge, but nothing serious or intractable.
>> >>>
>> >>> If there are no objections, I think it would be reasonable to merge
>> this
>> >>> code after this week's beta builds are published (after, in order to
>> avoid
>> >>> major changes during Summit / PyCon prep time).
>> >>>
>> >>> Thank you, Brian, for doing the planning and work needed to make this
>> >>> happen.  It was a lot of effort and is very highly appreciated.
>> >>>
>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com>
>> >>> wrote:
>> >>>>
>> >>>> Through several rebases, now all PRs are showing the RQ PRs on
>> Travis as
>> >>>> passing with pulp-smash. Several points of feedback have been
>> addressed.
>> >>>>
>> >>>> If you're interested in commenting on these PRs or trying them out,
>> >>>> please do. I hope to merge after the other taking system maintainers
>> @dalley
>> >>>> and @daviddavis have finished their testing/review and barring any
>> other
>> >>>> calls for delay or blocking concerns.
>> >>>>
>> >>>> If there are any questions, issues, or concerns, please reach out,
>> and
>> >>>> we can talk through them.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com
>> >
>> >>>> wrote:
>> >>>>>
>> >>>>> I put together a prototype and posted the PRs. I'm still working to
>> get
>> >>>>> Travis happy, but locally 100% of smash tests using these branches.
>> It's
>> >>>>> worked very reliably for me so far. There are no gaps in the pulp
>> feature
>> >>>>> set on top of RQ.
>> >>>>>
>> >>>>> I hope people test it out and give some feedback. See the commit
>> >>>>> messages for details on what was done. Here are the PRs:
>> >>>>>
>> >>>>> https://github.com/pulp/pulp/pull/3454
>> >>>>> https://github.com/pulp/pulp_file/pull/72
>> >>>>> https://github.com/pulp/devel/pull/146
>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>> >>>>>
>> >>>>> Feel free to send questions here or to the PR. Any feedback is
>> welcome.
>> >>>>>
>> >>>>> -Brian
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <mkovacik at redhat.com
>> >
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>> >>>>>> there's Fakeredis ;)
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> milan
>> >>>>>>
>> >>>>>>
>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>> bbouters at redhat.com>
>> >>>>>> wrote:
>> >>>>>> > Thanks for all the discussion both on list and on irc. After more
>> >>>>>> > investigation, it sounds like there are no feature gaps, but we
>> will
>> >>>>>> > need to
>> >>>>>> > incorporate this workaround to cancel a task that is already
>> >>>>>> > running.
>> >>>>>> >
>> >>>>>> > The feedback I've heard on the idea is that it's valuable and
>> looks
>> >>>>>> > feasible, but we won't really know until we prototype it a bit.
>> >>>>>> > Based on the
>> >>>>>> > technical outline in the previous email, I believe it can be
>> >>>>>> > prototyped in a
>> >>>>>> > day or two. I plan to do this soon, once I contribute to a few
>> other
>> >>>>>> > required-for-beta planning items first. I'll post my PR to see
>> what
>> >>>>>> > other
>> >>>>>> > think of the change, probably next week.
>> >>>>>> >
>> >>>>>> >
>> >>>>>> >
>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <dalley at redhat.com
>> >
>> >>>>>> > wrote:
>> >>>>>> >>
>> >>>>>> >> I meant in the sense that, what is the aftermath when it comes
>> back
>> >>>>>> >> online, and is it screwed up in ways that cause side effects.
>> >>>>>> >>
>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>> jaudet at redhat.com>
>> >>>>>> >> wrote:
>> >>>>>> >>>
>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the worker a
>> >>>>>> >>> > SIGINT,
>> >>>>>> >>> > it will finish the task and then stop processing new ones.
>> If
>> >>>>>> >>> > you send the
>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't think
>> it
>> >>>>>> >>> > gracefully
>> >>>>>> >>> > handles this circumstance.
>> >>>>>> >>>
>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch that
>> >>>>>> >>> signal.
>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>> >>>>>> >>>
>> >>>>>> >>> If one is looking for a way to gracefully, immediately kill an
>> RQ
>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
>> processes
>> >>>>>> >>> handle this signal in a hurried fashion. Semantically, this is
>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c
>> >>>>>> >>> sends
>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>> >>>>>> >>
>> >>>>>> >>
>> >>>>>> >
>> >>>>>> >
>> >>>>>> > _______________________________________________
>> >>>>>> > Pulp-dev mailing list
>> >>>>>> > Pulp-dev at redhat.com
>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>>>>> >
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> Pulp-dev mailing list
>> >> Pulp-dev at redhat.com
>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > Pulp-dev mailing list
>> > Pulp-dev at redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180514/479b4c8e/attachment.htm>