[Pulp-dev] Port Pulp3 to use RQ

Mon May 14 21:10:43 UTC 2018

Great, thanks.

David

On Mon, May 14, 2018 at 4:46 PM, Brian Bouterse <bbouters at redhat.com> wrote:

> If you are using the dev environment produced by the pulp/devel repo then
> doing a vagrant destroy and vagrant up is the easiest way. Just make sure
> you fetched all the latest changes from all the repos.
>
> If you want to port an existing environment into RQ, you should:
>
> 1. install RQ (the special commit version, see the current installation
> docs)
> 2. update the systemd files (see the examples in the docs)
> 3. restart your workers
>
> If there are issues during ^ steps reach out and we can help resolve them.
>
> On Mon, May 14, 2018 at 4:39 PM, David Davis <daviddavis at redhat.com>
> wrote:
>
>> Is there any way to convert an existing dev environment to use rq?
>>
>> Or do I just need to vagrant destroy and vagrant up?
>>
>>
>> David
>>
>> On Mon, May 14, 2018 at 4:02 PM, Daniel Alley <dalley at redhat.com> wrote:
>>
>>> And pulp_python
>>>
>>> On Mon, May 14, 2018 at 3:55 PM, Brian Bouterse <bbouters at redhat.com>
>>> wrote:
>>>
>>>> RQ is merged to pulp, pulp_file, pulp-smash, and devel. We also ported
>>>> and merged pulp_ansible. This will be released with beta 3 of core coming
>>>> out this Wednesday.
>>>>
>>>> If anyone runs into any issues please reach out via IRC or the mailing
>>>> list.
>>>>
>>>> On Mon, May 14, 2018 at 12:41 PM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>>
>>>>> There's been a slight change in schedule. Now we believe the lowest
>>>>> risk option is to merge today instead of tomorrow.
>>>>>
>>>>> We're finishing the latest rebase now, letting Travis tell us it's
>>>>> good, and then merging it. We'll send a final note to the list post merge.
>>>>>
>>>>> Thanks to everyone for helping out!
>>>>>
>>>>> On Mon, May 14, 2018 at 12:07 PM, Dana Walker <dawalker at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> +1 to advance notice, and +1 to @bmbouter and @dalley on the work,
>>>>>> review/testing, and blog post.
>>>>>>
>>>>>> Dana Walker
>>>>>>
>>>>>> Associate Software Engineer
>>>>>>
>>>>>> Red Hat
>>>>>>
>>>>>> <https://www.redhat.com>
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>> On Wed, May 9, 2018 at 12:20 PM, David Davis <daviddavis at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Great work on this. Also, thanks for announcing this on pulp-dev
>>>>>>> well in advance.
>>>>>>>
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>> On Wed, May 9, 2018 at 8:29 AM, Robin Chan <rchan at redhat.com> wrote:
>>>>>>>
>>>>>>>> dalley has learned how to do some debugging already, so maybe he can
>>>>>>>> look at doing a demo. Good suggestion, Kersom. It would be good to
>>>>>>>> link to in a blog post - and  also point out the good demo @bmbouter
>>>>>>>> put together for pulp 2.
>>>>>>>>
>>>>>>>> great job @dalley & @bmbouter on the blog post!
>>>>>>>>
>>>>>>>> On Wed, May 9, 2018 at 11:24 AM, Kersom <kersom at redhat.com> wrote:
>>>>>>>> > At the proper time, a demo about the Pulp 3 task system will be
>>>>>>>> very
>>>>>>>> > beneficial. I am thinking about something similar what it was
>>>>>>>> done for Pulp
>>>>>>>> > 2.
>>>>>>>> >
>>>>>>>> > Looking forward for this.
>>>>>>>> >
>>>>>>>> > Regards,
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, May 9, 2018 at 10:32 AM, Brian Bouterse <
>>>>>>>> bbouters at redhat.com> wrote:
>>>>>>>> >>
>>>>>>>> >> All PRs have Travis showing green and all necessary LGTMs. The
>>>>>>>> plan is to
>>>>>>>> >> merge next Tuesday the 15th, which means it will be in core Beta
>>>>>>>> 4.
>>>>>>>> >>
>>>>>>>> >> Yesterday, @dalley and I published a blog post which outlines
>>>>>>>> the change
>>>>>>>> >> for users along with a porting guide for plugins to port onto RQ
>>>>>>>> as well.
>>>>>>>> >>
>>>>>>>> >> https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/
>>>>>>>> >>
>>>>>>>> >> Thank you to everyone for the help, collaboration, and energy on
>>>>>>>> this
>>>>>>>> >> significant change.
>>>>>>>> >>
>>>>>>>> >> On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> I've finished my review and resolved all of the 'blocker'
>>>>>>>> issues that
>>>>>>>> >>> were uncovered during testing.  Overall, I'm highly confident
>>>>>>>> that this is a
>>>>>>>> >>> better path forwards than the continued use of Celery / Kombu.
>>>>>>>> There are a
>>>>>>>> >>> couple of outstanding edge cases to be resolved eventually,
>>>>>>>> which I plan to
>>>>>>>> >>> file as issues post-merge, but nothing serious or intractable.
>>>>>>>> >>>
>>>>>>>> >>> If there are no objections, I think it would be reasonable to
>>>>>>>> merge this
>>>>>>>> >>> code after this week's beta builds are published (after, in
>>>>>>>> order to avoid
>>>>>>>> >>> major changes during Summit / PyCon prep time).
>>>>>>>> >>>
>>>>>>>> >>> Thank you, Brian, for doing the planning and work needed to
>>>>>>>> make this
>>>>>>>> >>> happen.  It was a lot of effort and is very highly appreciated.
>>>>>>>> >>>
>>>>>>>> >>> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
>>>>>>>> bbouters at redhat.com>
>>>>>>>> >>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Through several rebases, now all PRs are showing the RQ PRs on
>>>>>>>> Travis as
>>>>>>>> >>>> passing with pulp-smash. Several points of feedback have been
>>>>>>>> addressed.
>>>>>>>> >>>>
>>>>>>>> >>>> If you're interested in commenting on these PRs or trying them
>>>>>>>> out,
>>>>>>>> >>>> please do. I hope to merge after the other taking system
>>>>>>>> maintainers @dalley
>>>>>>>> >>>> and @daviddavis have finished their testing/review and barring
>>>>>>>> any other
>>>>>>>> >>>> calls for delay or blocking concerns.
>>>>>>>> >>>>
>>>>>>>> >>>> If there are any questions, issues, or concerns, please reach
>>>>>>>> out, and
>>>>>>>> >>>> we can talk through them.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <
>>>>>>>> bbouters at redhat.com>
>>>>>>>> >>>> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> I put together a prototype and posted the PRs. I'm still
>>>>>>>> working to get
>>>>>>>> >>>>> Travis happy, but locally 100% of smash tests using these
>>>>>>>> branches. It's
>>>>>>>> >>>>> worked very reliably for me so far. There are no gaps in the
>>>>>>>> pulp feature
>>>>>>>> >>>>> set on top of RQ.
>>>>>>>> >>>>>
>>>>>>>> >>>>> I hope people test it out and give some feedback. See the
>>>>>>>> commit
>>>>>>>> >>>>> messages for details on what was done. Here are the PRs:
>>>>>>>> >>>>>
>>>>>>>> >>>>> https://github.com/pulp/pulp/pull/3454
>>>>>>>> >>>>> https://github.com/pulp/pulp_file/pull/72
>>>>>>>> >>>>> https://github.com/pulp/devel/pull/146
>>>>>>>> >>>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>>>>>> >>>>>
>>>>>>>> >>>>> Feel free to send questions here or to the PR. Any feedback
>>>>>>>> is welcome.
>>>>>>>> >>>>>
>>>>>>>> >>>>> -Brian
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <
>>>>>>>> mkovacik at redhat.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/
>>>>>>>> esp.
>>>>>>>> >>>>>> there's Fakeredis ;)
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> --
>>>>>>>> >>>>>> milan
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <
>>>>>>>> bbouters at redhat.com>
>>>>>>>> >>>>>> wrote:
>>>>>>>> >>>>>> > Thanks for all the discussion both on list and on irc.
>>>>>>>> After more
>>>>>>>> >>>>>> > investigation, it sounds like there are no feature gaps,
>>>>>>>> but we will
>>>>>>>> >>>>>> > need to
>>>>>>>> >>>>>> > incorporate this workaround to cancel a task that is
>>>>>>>> already
>>>>>>>> >>>>>> > running.
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> > The feedback I've heard on the idea is that it's valuable
>>>>>>>> and looks
>>>>>>>> >>>>>> > feasible, but we won't really know until we prototype it a
>>>>>>>> bit.
>>>>>>>> >>>>>> > Based on the
>>>>>>>> >>>>>> > technical outline in the previous email, I believe it can
>>>>>>>> be
>>>>>>>> >>>>>> > prototyped in a
>>>>>>>> >>>>>> > day or two. I plan to do this soon, once I contribute to a
>>>>>>>> few other
>>>>>>>> >>>>>> > required-for-beta planning items first. I'll post my PR to
>>>>>>>> see what
>>>>>>>> >>>>>> > other
>>>>>>>> >>>>>> > think of the change, probably next week.
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <
>>>>>>>> dalley at redhat.com>
>>>>>>>> >>>>>> > wrote:
>>>>>>>> >>>>>> >>
>>>>>>>> >>>>>> >> I meant in the sense that, what is the aftermath when it
>>>>>>>> comes back
>>>>>>>> >>>>>> >> online, and is it screwed up in ways that cause side
>>>>>>>> effects.
>>>>>>>> >>>>>> >>
>>>>>>>> >>>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <
>>>>>>>> jaudet at redhat.com>
>>>>>>>> >>>>>> >> wrote:
>>>>>>>> >>>>>> >>>
>>>>>>>> >>>>>> >>> > RQ does not support revoking tasks.  If you send the
>>>>>>>> worker a
>>>>>>>> >>>>>> >>> > SIGINT,
>>>>>>>> >>>>>> >>> > it will finish the task and then stop processing new
>>>>>>>> ones.  If
>>>>>>>> >>>>>> >>> > you send the
>>>>>>>> >>>>>> >>> > worker SIGKILL, it will stop immediately, but I don't
>>>>>>>> think it
>>>>>>>> >>>>>> >>> > gracefully
>>>>>>>> >>>>>> >>> > handles this circumstance.
>>>>>>>> >>>>>> >>>
>>>>>>>> >>>>>> >>> Nothing handles SIGKILL gracefully. Processes can't
>>>>>>>> catch that
>>>>>>>> >>>>>> >>> signal.
>>>>>>>> >>>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>>>>>> >>>>>> >>>
>>>>>>>> >>>>>> >>> If one is looking for a way to gracefully, immediately
>>>>>>>> kill an RQ
>>>>>>>> >>>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many
>>>>>>>> processes
>>>>>>>> >>>>>> >>> handle this signal in a hurried fashion. Semantically,
>>>>>>>> this is
>>>>>>>> >>>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal
>>>>>>>> (Ctrl+c
>>>>>>>> >>>>>> >>> sends
>>>>>>>> >>>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>>>>>> >>>>>> >>
>>>>>>>> >>>>>> >>
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>> > _______________________________________________
>>>>>>>> >>>>>> > Pulp-dev mailing list
>>>>>>>> >>>>>> > Pulp-dev at redhat.com
>>>>>>>> >>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>> >>>>>> >
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>
>>>>>>>> >>>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> _______________________________________________
>>>>>>>> >> Pulp-dev mailing list
>>>>>>>> >> Pulp-dev at redhat.com
>>>>>>>> >> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > Pulp-dev mailing list
>>>>>>>> > Pulp-dev at redhat.com
>>>>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>> >
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180514/84917f91/attachment.htm>