[Pulp-dev] Port Pulp3 to use RQ

Wed May 9 16:20:25 UTC 2018

Great work on this. Also, thanks for announcing this on pulp-dev well in
advance.

David

On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:

> dalley has learned how to do some debugging already, so maybe he can
> look at doing a demo. Good suggestion, Kersom. It would be good to
> link to in a blog post - and  also point out the good demo @bmbouter
> put together for pulp 2.
>
> great job @dalley & @bmbouter on the blog post!
>
> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
> > At the proper time, a demo about the Pulp 3 task system will be very
> > beneficial. I am thinking about something similar what it was done for
> Pulp
> > 2.
> >
> > Looking forward for this.
> >
> > Regards,
> >
> >
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <bbouters at redhat.com>
> wrote:
> >>
> >> All PRs have Travis showing green and all necessary LGTMs. The plan is
> to
> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
> >>
> >> Yesterday, @dalley and I published a blog post which outlines the change
> >> for users along with a porting guide for plugins to port onto RQ as
> well.
> >>
> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
> >>
> >> Thank you to everyone for the help, collaboration, and energy on this
> >> significant change.
> >>
> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com> wrote:
> >>>
> >>> I've finished my review and resolved all of the 'blocker' issues that
> >>> were uncovered during testing.  Overall, I'm highly confident that
> this is a
> >>> better path forwards than the continued use of Celery / Kombu.  There
> are a
> >>> couple of outstanding edge cases to be resolved eventually, which I
> plan to
> >>> file as issues post-merge, but nothing serious or intractable.
> >>>
> >>> If there are no objections, I think it would be reasonable to merge
> this
> >>> code after this week's beta builds are published (after, in order to
> avoid
> >>> major changes during Summit / PyCon prep time).
> >>>
> >>> Thank you, Brian, for doing the planning and work needed to make this
> >>> happen.  It was a lot of effort and is very highly appreciated.
> >>>
> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com>
> >>> wrote:
> >>>>
> >>>> Through several rebases, now all PRs are showing the RQ PRs on Travis
> as
> >>>> passing with pulp-smash. Several points of feedback have been
> addressed.
> >>>>
> >>>> If you're interested in commenting on these PRs or trying them out,
> >>>> please do. I hope to merge after the other taking system maintainers
> @dalley
> >>>> and @daviddavis have finished their testing/review and barring any
> other
> >>>> calls for delay or blocking concerns.
> >>>>
> >>>> If there are any questions, issues, or concerns, please reach out, and
> >>>> we can talk through them.
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com>
> >>>> wrote:
> >>>>>
> >>>>> I put together a prototype and posted the PRs. I'm still working to
> get
> >>>>> Travis happy, but locally 100% of smash tests using these branches.
> It's
> >>>>> worked very reliably for me so far. There are no gaps in the pulp
> feature
> >>>>> set on top of RQ.
> >>>>>
> >>>>> I hope people test it out and give some feedback. See the commit
> >>>>> messages for details on what was done. Here are the PRs:
> >>>>>
> >>>>> https://github.com/pulp/pulp/pull/3454
> >>>>> https://github.com/pulp/pulp_file/pull/72
> >>>>> https://github.com/pulp/devel/pull/146
> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
> >>>>>
> >>>>> Feel free to send questions here or to the PR. Any feedback is
> welcome.
> >>>>>
> >>>>> -Brian
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <mkovacik at redhat.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
> >>>>>> there's Fakeredis ;)
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> milan
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
> bbouters at redhat.com>
> >>>>>> wrote:
> >>>>>> > Thanks for all the discussion both on list and on irc. After more
> >>>>>> > investigation, it sounds like there are no feature gaps, but we
> will
> >>>>>> > need to
> >>>>>> > incorporate this workaround to cancel a task that is already
> >>>>>> > running.
> >>>>>> >
> >>>>>> > The feedback I've heard on the idea is that it's valuable and
> looks
> >>>>>> > feasible, but we won't really know until we prototype it a bit.
> >>>>>> > Based on the
> >>>>>> > technical outline in the previous email, I believe it can be
> >>>>>> > prototyped in a
> >>>>>> > day or two. I plan to do this soon, once I contribute to a few
> other
> >>>>>> > required-for-beta planning items first. I'll post my PR to see
> what
> >>>>>> > other
> >>>>>> > think of the change, probably next week.
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <dalley at redhat.com>
> >>>>>> > wrote:
> >>>>>> >>
> >>>>>> >> I meant in the sense that, what is the aftermath when it comes
> back
> >>>>>> >> online, and is it screwed up in ways that cause side effects.
> >>>>>> >>
> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <jaudet at redhat.com
> >
> >>>>>> >> wrote:
> >>>>>> >>>
> >>>>>> >>> > RQ does not support revoking tasks.  If you send the worker a
> >>>>>> >>> > SIGINT,
> >>>>>> >>> > it will finish the task and then stop processing new ones.  If
> >>>>>> >>> > you send the
> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't think it
> >>>>>> >>> > gracefully
> >>>>>> >>> > handles this circumstance.
> >>>>>> >>>
> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch that
> >>>>>> >>> signal.
> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
> >>>>>> >>>
> >>>>>> >>> If one is looking for a way to gracefully, immediately kill an
> RQ
> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
> processes
> >>>>>> >>> handle this signal in a hurried fashion. Semantically, this is
> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c
> >>>>>> >>> sends
> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
> >>>>>> >>
> >>>>>> >>
> >>>>>> >
> >>>>>> >
> >>>>>> > _______________________________________________
> >>>>>> > Pulp-dev mailing list
> >>>>>> > Pulp-dev at redhat.com
> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >>>>>> >
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> Pulp-dev mailing list
> >> Pulp-dev at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>
> >
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180509/8e0652a1/attachment.htm>