[Pulp-dev] Port Pulp3 to use RQ

David Davis daviddavis at redhat.com
Mon May 14 20:39:57 UTC 2018


Is there any way to convert an existing dev environment to use rq?

Or do I just need to vagrant destroy and vagrant up?


David

On Mon, May 14, 2018 at 4:02 PM, Daniel Alley <dalley at redhat.com> wrote:

> And pulp_python
>
> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
>> and merged pulp_ansible. This will be released with beta 3 of core coming
>> out this Wednesday.
>>
>> If anyone runs into any issues please reach out via IRC or the mailing
>> list.
>>
>> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> There's been a slight change in schedule. Now we believe the lowest risk
>>> option is to merge today instead of tomorrow.
>>>
>>> We're finishing the latest rebase now, letting Travis tell us it's good,
>>> and then merging it. We'll send a final note to the list post merge.
>>>
>>> Thanks to everyone for helping out!
>>>
>>> On Mon, May 14, 2018 at 12:07 PM, Dana Walker <dawalker at redhat.com>
>>> wrote:
>>>
>>>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>>>> review/testing, and blog post.
>>>>
>>>> Dana Walker
>>>>
>>>> Associate Software Engineer
>>>>
>>>> Red Hat
>>>>
>>>> <https://www.redhat.com>
>>>> <https://red.ht/sig>
>>>>
>>>> On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com>
>>>> wrote:
>>>>
>>>>> Great work on this. Also, thanks for announcing this on pulp-dev well
>>>>> in advance.
>>>>>
>>>>>
>>>>> David
>>>>>
>>>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:
>>>>>
>>>>>> dalley has learned how to do some debugging already, so maybe he can
>>>>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>>>>> link to in a blog post - and  also point out the good demo @bmbouter
>>>>>> put together for pulp 2.
>>>>>>
>>>>>> great job @dalley & @bmbouter on the blog post!
>>>>>>
>>>>>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>>>>>> > At the proper time, a demo about the Pulp 3 task system will be very
>>>>>> > beneficial. I am thinking about something similar what it was done
>>>>>> for Pulp
>>>>>> > 2.
>>>>>> >
>>>>>> > Looking forward for this.
>>>>>> >
>>>>>> > Regards,
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
>>>>>> bbouters at redhat.com> wrote:
>>>>>> >>
>>>>>> >> All PRs have Travis showing green and all necessary LGTMs. The
>>>>>> plan is to
>>>>>> >> merge next Tuesday the 15th, which means it will be in core Beta 4.
>>>>>> >>
>>>>>> >> Yesterday, @dalley and I published a blog post which outlines the
>>>>>> change
>>>>>> >> for users along with a porting guide for plugins to port onto RQ
>>>>>> as well.
>>>>>> >>
>>>>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>>>>> >>
>>>>>> >> Thank you to everyone for the help, collaboration, and energy on
>>>>>> this
>>>>>> >> significant change.
>>>>>> >>
>>>>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> I've finished my review and resolved all of the 'blocker' issues
>>>>>> that
>>>>>> >>> were uncovered during testing.  Overall, I'm highly confident
>>>>>> that this is a
>>>>>> >>> better path forwards than the continued use of Celery / Kombu.
>>>>>> There are a
>>>>>> >>> couple of outstanding edge cases to be resolved eventually, which
>>>>>> I plan to
>>>>>> >>> file as issues post-merge, but nothing serious or intractable.
>>>>>> >>>
>>>>>> >>> If there are no objections, I think it would be reasonable to
>>>>>> merge this
>>>>>> >>> code after this week's beta builds are published (after, in order
>>>>>> to avoid
>>>>>> >>> major changes during Summit / PyCon prep time).
>>>>>> >>>
>>>>>> >>> Thank you, Brian, for doing the planning and work needed to make
>>>>>> this
>>>>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>>>>> >>>
>>>>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>>>>> bbouters at redhat.com>
>>>>>> >>> wrote:
>>>>>> >>>>
>>>>>> >>>> Through several rebases, now all PRs are showing the RQ PRs on
>>>>>> Travis as
>>>>>> >>>> passing with pulp-smash. Several points of feedback have been
>>>>>> addressed.
>>>>>> >>>>
>>>>>> >>>> If you're interested in commenting on these PRs or trying them
>>>>>> out,
>>>>>> >>>> please do. I hope to merge after the other taking system
>>>>>> maintainers @dalley
>>>>>> >>>> and @daviddavis have finished their testing/review and barring
>>>>>> any other
>>>>>> >>>> calls for delay or blocking concerns.
>>>>>> >>>>
>>>>>> >>>> If there are any questions, issues, or concerns, please reach
>>>>>> out, and
>>>>>> >>>> we can talk through them.
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>>>>> bbouters at redhat.com>
>>>>>> >>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> I put together a prototype and posted the PRs. I'm still
>>>>>> working to get
>>>>>> >>>>> Travis happy, but locally 100% of smash tests using these
>>>>>> branches. It's
>>>>>> >>>>> worked very reliably for me so far. There are no gaps in the
>>>>>> pulp feature
>>>>>> >>>>> set on top of RQ.
>>>>>> >>>>>
>>>>>> >>>>> I hope people test it out and give some feedback. See the commit
>>>>>> >>>>> messages for details on what was done. Here are the PRs:
>>>>>> >>>>>
>>>>>> >>>>> https://github.com/pulp/pulp/pull/3454
>>>>>> >>>>> https://github.com/pulp/pulp_file/pull/72
>>>>>> >>>>> https://github.com/pulp/devel/pull/146
>>>>>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>>>> >>>>>
>>>>>> >>>>> Feel free to send questions here or to the PR. Any feedback is
>>>>>> welcome.
>>>>>> >>>>>
>>>>>> >>>>> -Brian
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>>>>> mkovacik at redhat.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/
>>>>>> esp.
>>>>>> >>>>>> there's Fakeredis ;)
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> --
>>>>>> >>>>>> milan
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>>>>>> bbouters at redhat.com>
>>>>>> >>>>>> wrote:
>>>>>> >>>>>> > Thanks for all the discussion both on list and on irc. After
>>>>>> more
>>>>>> >>>>>> > investigation, it sounds like there are no feature gaps, but
>>>>>> we will
>>>>>> >>>>>> > need to
>>>>>> >>>>>> > incorporate this workaround to cancel a task that is already
>>>>>> >>>>>> > running.
>>>>>> >>>>>> >
>>>>>> >>>>>> > The feedback I've heard on the idea is that it's valuable
>>>>>> and looks
>>>>>> >>>>>> > feasible, but we won't really know until we prototype it a
>>>>>> bit.
>>>>>> >>>>>> > Based on the
>>>>>> >>>>>> > technical outline in the previous email, I believe it can be
>>>>>> >>>>>> > prototyped in a
>>>>>> >>>>>> > day or two. I plan to do this soon, once I contribute to a
>>>>>> few other
>>>>>> >>>>>> > required-for-beta planning items first. I'll post my PR to
>>>>>> see what
>>>>>> >>>>>> > other
>>>>>> >>>>>> > think of the change, probably next week.
>>>>>> >>>>>> >
>>>>>> >>>>>> >
>>>>>> >>>>>> >
>>>>>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <
>>>>>> dalley at redhat.com>
>>>>>> >>>>>> > wrote:
>>>>>> >>>>>> >>
>>>>>> >>>>>> >> I meant in the sense that, what is the aftermath when it
>>>>>> comes back
>>>>>> >>>>>> >> online, and is it screwed up in ways that cause side
>>>>>> effects.
>>>>>> >>>>>> >>
>>>>>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>>>>>> jaudet at redhat.com>
>>>>>> >>>>>> >> wrote:
>>>>>> >>>>>> >>>
>>>>>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the
>>>>>> worker a
>>>>>> >>>>>> >>> > SIGINT,
>>>>>> >>>>>> >>> > it will finish the task and then stop processing new
>>>>>> ones.  If
>>>>>> >>>>>> >>> > you send the
>>>>>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't
>>>>>> think it
>>>>>> >>>>>> >>> > gracefully
>>>>>> >>>>>> >>> > handles this circumstance.
>>>>>> >>>>>> >>>
>>>>>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch
>>>>>> that
>>>>>> >>>>>> >>> signal.
>>>>>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>>>> >>>>>> >>>
>>>>>> >>>>>> >>> If one is looking for a way to gracefully, immediately
>>>>>> kill an RQ
>>>>>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
>>>>>> processes
>>>>>> >>>>>> >>> handle this signal in a hurried fashion. Semantically,
>>>>>> this is
>>>>>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal
>>>>>> (Ctrl+c
>>>>>> >>>>>> >>> sends
>>>>>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>>>> >>>>>> >>
>>>>>> >>>>>> >>
>>>>>> >>>>>> >
>>>>>> >>>>>> >
>>>>>> >>>>>> > _______________________________________________
>>>>>> >>>>>> > Pulp-dev mailing list
>>>>>> >>>>>> > Pulp-dev at redhat.com
>>>>>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>> >>>>>> >
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >> _______________________________________________
>>>>>> >> Pulp-dev mailing list
>>>>>> >> Pulp-dev at redhat.com
>>>>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Pulp-dev mailing list
>>>>>> > Pulp-dev at redhat.com
>>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>> >
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180514/1f1ead6a/attachment.htm>


More information about the Pulp-dev mailing list