[Pulp-dev] Port Pulp3 to use RQ

[David Davis]

+1. Thank you @bmbouter and @dalley for working on this.


David

On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com> wrote:

> I've finished my review and resolved all of the 'blocker' issues that were
> uncovered during testing.  Overall, I'm highly confident that this is a
> better path forwards than the continued use of Celery / Kombu.  There are a
> couple of outstanding edge cases to be resolved eventually, which I plan to
> file as issues post-merge, but nothing serious or intractable.
>
> If there are no objections, I think it would be reasonable to merge this
> code after this week's beta builds are published (after, in order to avoid
> major changes during Summit / PyCon prep time).
>
> Thank you, Brian, for doing the planning and work needed to make this
> happen.  It was a lot of effort and is very highly appreciated.
>
> On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> Through several rebases, now all PRs are showing the RQ PRs on Travis as
>> passing with pulp-smash. Several points of feedback have been addressed.
>>
>> If you're interested in commenting on these PRs or trying them out,
>> please do. I hope to merge after the other taking system maintainers
>> @dalley and @daviddavis have finished their testing/review and barring any
>> other calls for delay or blocking concerns.
>>
>> If there are any questions, issues, or concerns, please reach out, and we
>> can talk through them.
>>
>>
>>
>> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com>
>> wrote:
>>
>>> I put together a prototype and posted the PRs. I'm still working to get
>>> Travis happy, but locally 100% of smash tests using these branches. It's
>>> worked very reliably for me so far. There are no gaps in the pulp feature
>>> set on top of RQ.
>>>
>>> I hope people test it out and give some feedback. See the commit
>>> messages for details on what was done. Here are the PRs:
>>>
>>> https://github.com/pulp/pulp/pull/3454
>>> https://github.com/pulp/pulp_file/pull/72
>>> https://github.com/pulp/devel/pull/146
>>> https://github.com/PulpQE/pulp-smash/pull/960
>>>
>>> Feel free to send questions here or to the PR. Any feedback is welcome.
>>>
>>> -Brian
>>>
>>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <mkovacik at redhat.com>
>>> wrote:
>>>
>>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>>> there's Fakeredis ;)
>>>>
>>>>
>>>> --
>>>> milan
>>>>
>>>>
>>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <bbouters at redhat.com>
>>>> wrote:
>>>> > Thanks for all the discussion both on list and on irc. After more
>>>> > investigation, it sounds like there are no feature gaps, but we will
>>>> need to
>>>> > incorporate this workaround to cancel a task that is already running.
>>>> >
>>>> > The feedback I've heard on the idea is that it's valuable and looks
>>>> > feasible, but we won't really know until we prototype it a bit. Based
>>>> on the
>>>> > technical outline in the previous email, I believe it can be
>>>> prototyped in a
>>>> > day or two. I plan to do this soon, once I contribute to a few other
>>>> > required-for-beta planning items first. I'll post my PR to see what
>>>> other
>>>> > think of the change, probably next week.
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <dalley at redhat.com>
>>>> wrote:
>>>> >>
>>>> >> I meant in the sense that, what is the aftermath when it comes back
>>>> >> online, and is it screwed up in ways that cause side effects.
>>>> >>
>>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <jaudet at redhat.com>
>>>> wrote:
>>>> >>>
>>>> >>> > RQ does not support revoking tasks.  If you send the worker a
>>>> SIGINT,
>>>> >>> > it will finish the task and then stop processing new ones.  If
>>>> you send the
>>>> >>> > worker SIGKILL, it will stop immediately, but I don't think it
>>>> gracefully
>>>> >>> > handles this circumstance.
>>>> >>>
>>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch that
>>>> signal.
>>>> >>> `kill -9 $pid` sends SIGKILL.
>>>> >>>
>>>> >>> If one is looking for a way to gracefully, immediately kill an RQ
>>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many processes
>>>> >>> handle this signal in a hurried fashion. Semantically, this is
>>>> >>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c sends
>>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Pulp-dev mailing list
>>>> > Pulp-dev at redhat.com
>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>> >
>>>>
>>>
>>>
>>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180508/6ed0b6a9/attachment.htm>