[Pulp-dev] Port Pulp3 to use RQ

Mon May 14 20:02:30 UTC 2018

And pulp_python

On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported and
> merged pulp_ansible. This will be released with beta 3 of core coming out
> this Wednesday.
>
> If anyone runs into any issues please reach out via IRC or the mailing
> list.
>
> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> There's been a slight change in schedule. Now we believe the lowest risk
>> option is to merge today instead of tomorrow.
>>
>> We're finishing the latest rebase now, letting Travis tell us it's good,
>> and then merging it. We'll send a final note to the list post merge.
>>
>> Thanks to everyone for helping out!
>>
>> On Mon, May 14, 2018 at 12:07 PM, Dana Walker <dawalker at redhat.com>
>> wrote:
>>
>>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>>> review/testing, and blog post.
>>>
>>> Dana Walker
>>>
>>> Associate Software Engineer
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com>
>>> <https://red.ht/sig>
>>>
>>> On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com>
>>> wrote:
>>>
>>>> Great work on this. Also, thanks for announcing this on pulp-dev well
>>>> in advance.
>>>>
>>>>
>>>> David
>>>>
>>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:
>>>>
>>>>> dalley has learned how to do some debugging already, so maybe he can
>>>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>>>> link to in a blog post - and  also point out the good demo @bmbouter
>>>>> put together for pulp 2.
>>>>>
>>>>> great job @dalley & @bmbouter on the blog post!
>>>>>
>>>>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>>>>> > At the proper time, a demo about the Pulp 3 task system will be very
>>>>> > beneficial. I am thinking about something similar what it was done
>>>>> for Pulp
>>>>> > 2.
>>>>> >
>>>>> > Looking forward for this.
>>>>> >
>>>>> > Regards,
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <bbouters at redhat.com>
>>>>> wrote:
>>>>> >>
>>>>> >> All PRs have Travis showing green and all necessary LGTMs. The plan
>>>>> is to
>>>>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>>>>> >>
>>>>> >> Yesterday, @dalley and I published a blog post which outlines the
>>>>> change
>>>>> >> for users along with a porting guide for plugins to port onto RQ as
>>>>> well.
>>>>> >>
>>>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>>>> >>
>>>>> >> Thank you to everyone for the help, collaboration, and energy on
>>>>> this
>>>>> >> significant change.
>>>>> >>
>>>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> I've finished my review and resolved all of the 'blocker' issues
>>>>> that
>>>>> >>> were uncovered during testing.  Overall, I'm highly confident that
>>>>> this is a
>>>>> >>> better path forwards than the continued use of Celery / Kombu.
>>>>> There are a
>>>>> >>> couple of outstanding edge cases to be resolved eventually, which
>>>>> I plan to
>>>>> >>> file as issues post-merge, but nothing serious or intractable.
>>>>> >>>
>>>>> >>> If there are no objections, I think it would be reasonable to
>>>>> merge this
>>>>> >>> code after this week's beta builds are published (after, in order
>>>>> to avoid
>>>>> >>> major changes during Summit / PyCon prep time).
>>>>> >>>
>>>>> >>> Thank you, Brian, for doing the planning and work needed to make
>>>>> this
>>>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>>>> >>>
>>>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>>>> bbouters at redhat.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> Through several rebases, now all PRs are showing the RQ PRs on
>>>>> Travis as
>>>>> >>>> passing with pulp-smash. Several points of feedback have been
>>>>> addressed.
>>>>> >>>>
>>>>> >>>> If you're interested in commenting on these PRs or trying them
>>>>> out,
>>>>> >>>> please do. I hope to merge after the other taking system
>>>>> maintainers @dalley
>>>>> >>>> and @daviddavis have finished their testing/review and barring
>>>>> any other
>>>>> >>>> calls for delay or blocking concerns.
>>>>> >>>>
>>>>> >>>> If there are any questions, issues, or concerns, please reach
>>>>> out, and
>>>>> >>>> we can talk through them.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>>>> bbouters at redhat.com>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> I put together a prototype and posted the PRs. I'm still working
>>>>> to get
>>>>> >>>>> Travis happy, but locally 100% of smash tests using these
>>>>> branches. It's
>>>>> >>>>> worked very reliably for me so far. There are no gaps in the
>>>>> pulp feature
>>>>> >>>>> set on top of RQ.
>>>>> >>>>>
>>>>> >>>>> I hope people test it out and give some feedback. See the commit
>>>>> >>>>> messages for details on what was done. Here are the PRs:
>>>>> >>>>>
>>>>> >>>>> https://github.com/pulp/pulp/pull/3454
>>>>> >>>>> https://github.com/pulp/pulp_file/pull/72
>>>>> >>>>> https://github.com/pulp/devel/pull/146
>>>>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>>> >>>>>
>>>>> >>>>> Feel free to send questions here or to the PR. Any feedback is
>>>>> welcome.
>>>>> >>>>>
>>>>> >>>>> -Brian
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>>>> mkovacik at redhat.com>
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>>>> >>>>>> there's Fakeredis ;)
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> milan
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>>>>> bbouters at redhat.com>
>>>>> >>>>>> wrote:
>>>>> >>>>>> > Thanks for all the discussion both on list and on irc. After
>>>>> more
>>>>> >>>>>> > investigation, it sounds like there are no feature gaps, but
>>>>> we will
>>>>> >>>>>> > need to
>>>>> >>>>>> > incorporate this workaround to cancel a task that is already
>>>>> >>>>>> > running.
>>>>> >>>>>> >
>>>>> >>>>>> > The feedback I've heard on the idea is that it's valuable and
>>>>> looks
>>>>> >>>>>> > feasible, but we won't really know until we prototype it a
>>>>> bit.
>>>>> >>>>>> > Based on the
>>>>> >>>>>> > technical outline in the previous email, I believe it can be
>>>>> >>>>>> > prototyped in a
>>>>> >>>>>> > day or two. I plan to do this soon, once I contribute to a
>>>>> few other
>>>>> >>>>>> > required-for-beta planning items first. I'll post my PR to
>>>>> see what
>>>>> >>>>>> > other
>>>>> >>>>>> > think of the change, probably next week.
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <
>>>>> dalley at redhat.com>
>>>>> >>>>>> > wrote:
>>>>> >>>>>> >>
>>>>> >>>>>> >> I meant in the sense that, what is the aftermath when it
>>>>> comes back
>>>>> >>>>>> >> online, and is it screwed up in ways that cause side effects.
>>>>> >>>>>> >>
>>>>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>>>>> jaudet at redhat.com>
>>>>> >>>>>> >> wrote:
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the
>>>>> worker a
>>>>> >>>>>> >>> > SIGINT,
>>>>> >>>>>> >>> > it will finish the task and then stop processing new
>>>>> ones.  If
>>>>> >>>>>> >>> > you send the
>>>>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't
>>>>> think it
>>>>> >>>>>> >>> > gracefully
>>>>> >>>>>> >>> > handles this circumstance.
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch
>>>>> that
>>>>> >>>>>> >>> signal.
>>>>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> If one is looking for a way to gracefully, immediately kill
>>>>> an RQ
>>>>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
>>>>> processes
>>>>> >>>>>> >>> handle this signal in a hurried fashion. Semantically, this
>>>>> is
>>>>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal
>>>>> (Ctrl+c
>>>>> >>>>>> >>> sends
>>>>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>>> >>>>>> >>
>>>>> >>>>>> >>
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> > _______________________________________________
>>>>> >>>>>> > Pulp-dev mailing list
>>>>> >>>>>> > Pulp-dev at redhat.com
>>>>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>> >>>>>> >
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> Pulp-dev mailing list
>>>>> >> Pulp-dev at redhat.com
>>>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>> >>
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > Pulp-dev mailing list
>>>>> > Pulp-dev at redhat.com
>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>> >
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180514/d58614b8/attachment.htm>