[Pulp-dev] Port Pulp3 to use RQ

[Daniel Alley]

I've finished my review and resolved all of the 'blocker' issues that were
uncovered during testing.  Overall, I'm highly confident that this is a
better path forwards than the continued use of Celery / Kombu.  There are a
couple of outstanding edge cases to be resolved eventually, which I plan to
file as issues post-merge, but nothing serious or intractable.

If there are no objections, I think it would be reasonable to merge this
code after this week's beta builds are published (after, in order to avoid
major changes during Summit / PyCon prep time).

Thank you, Brian, for doing the planning and work needed to make this
happen.  It was a lot of effort and is very highly appreciated.

On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com> wrote:

> Through several rebases, now all PRs are showing the RQ PRs on Travis as
> passing with pulp-smash. Several points of feedback have been addressed.
>
> If you're interested in commenting on these PRs or trying them out, please
> do. I hope to merge after the other taking system maintainers @dalley and
> @daviddavis have finished their testing/review and barring any other calls
> for delay or blocking concerns.
>
> If there are any questions, issues, or concerns, please reach out, and we
> can talk through them.
>
>
>
> On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse <bbouters at redhat.com>
> wrote:
>
>> I put together a prototype and posted the PRs. I'm still working to get
>> Travis happy, but locally 100% of smash tests using these branches. It's
>> worked very reliably for me so far. There are no gaps in the pulp feature
>> set on top of RQ.
>>
>> I hope people test it out and give some feedback. See the commit messages
>> for details on what was done. Here are the PRs:
>>
>> https://github.com/pulp/pulp/pull/3454
>> https://github.com/pulp/pulp_file/pull/72
>> https://github.com/pulp/devel/pull/146
>> https://github.com/PulpQE/pulp-smash/pull/960
>>
>> Feel free to send questions here or to the PR. Any feedback is welcome.
>>
>> -Brian
>>
>>
>>
>>
>> On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik <mkovacik at redhat.com>
>> wrote:
>>
>>> +1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
>>> there's Fakeredis ;)
>>>
>>>
>>> --
>>> milan
>>>
>>>
>>> On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse <bbouters at redhat.com>
>>> wrote:
>>> > Thanks for all the discussion both on list and on irc. After more
>>> > investigation, it sounds like there are no feature gaps, but we will
>>> need to
>>> > incorporate this workaround to cancel a task that is already running.
>>> >
>>> > The feedback I've heard on the idea is that it's valuable and looks
>>> > feasible, but we won't really know until we prototype it a bit. Based
>>> on the
>>> > technical outline in the previous email, I believe it can be
>>> prototyped in a
>>> > day or two. I plan to do this soon, once I contribute to a few other
>>> > required-for-beta planning items first. I'll post my PR to see what
>>> other
>>> > think of the change, probably next week.
>>> >
>>> >
>>> >
>>> > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley <dalley at redhat.com>
>>> wrote:
>>> >>
>>> >> I meant in the sense that, what is the aftermath when it comes back
>>> >> online, and is it screwed up in ways that cause side effects.
>>> >>
>>> >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet <jaudet at redhat.com>
>>> wrote:
>>> >>>
>>> >>> > RQ does not support revoking tasks.  If you send the worker a
>>> SIGINT,
>>> >>> > it will finish the task and then stop processing new ones.  If you
>>> send the
>>> >>> > worker SIGKILL, it will stop immediately, but I don't think it
>>> gracefully
>>> >>> > handles this circumstance.
>>> >>>
>>> >>> Nothing handles SIGKILL gracefully. Processes can't catch that
>>> signal.
>>> >>> `kill -9 $pid` sends SIGKILL.
>>> >>>
>>> >>> If one is looking for a way to gracefully, immediately kill an RQ
>>> >>> worker, then SIGTERM may do the trick. Anecdotally, many processes
>>> >>> handle this signal in a hurried fashion. Semantically, this is
>>> >>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c sends
>>> >>> SIGINT), whereas SIGTERM is the "termination signal."
>>> >>
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Pulp-dev mailing list
>>> > Pulp-dev at redhat.com
>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180507/a5193f38/attachment.htm>