[Pulp-dev] Port Pulp3 to use RQ
Brian Bouterse
bbouters at redhat.com
Tue May 8 14:27:34 UTC 2018
Here is a look at what the Pulp2 -> Pulp3 necessary things would be w.r.t
this change. These could also be automated.
1. Empty the Pulp system of all of it's tasks and stop all Pulp services.
2. Uninstall RabbitMQ or Qpid if its only purpose was to serve the Pulp
tasking system. (Satellite uses Qpid in other ways so it would likely keep
it in the architecture for other purposes).
3. Uninstall Celery/kombu/billiard/py-amqp (the whole celery stack
effectively)
4. upgrade the bits to Pulp3. This will also bring RQ with it automatically.
5. Install Redis as a new service in your infra
6. Replace the systemd files for the pulp_workers and
pulp_resource_manager. This causes systemd to start RQ instead of Celery.
7. [optional] Configure Redis auth/ssl and configure Pulp's settings file
to match if that is part of your environment.
Questions/ideas/concerns are welcome.
On Tue, May 8, 2018 at 8:48 AM, Bryan Kearney <bkearney at redhat.com> wrote:
> what does this look like for upgrading from Pulp2 to Pulp3?
>
> -- bk
>
> On 05/08/2018 05:34 AM, David Davis wrote:
> > +1. Thank you @bmbouter and @dalley for working on this.
> >
> >
> > David
> >
> > On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com
> > <mailto:dalley at redhat.com>> wrote:
> >
> > I've finished my review and resolved all of the 'blocker' issues
> > that were uncovered during testing. Overall, I'm highly confident
> > that this is a better path forwards than the continued use of Celery
> > / Kombu. There are a couple of outstanding edge cases to be
> > resolved eventually, which I plan to file as issues post-merge, but
> > nothing serious or intractable.
> >
> > If there are no objections, I think it would be reasonable to merge
> > this code after this week's beta builds are published (after, in
> > order to avoid major changes during Summit / PyCon prep time).
> >
> > Thank you, Brian, for doing the planning and work needed to make
> > this happen. It was a lot of effort and is very highly appreciated.
> >
> > On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <bbouters at redhat.com
> > <mailto:bbouters at redhat.com>> wrote:
> >
> > Through several rebases, now all PRs are showing the RQ PRs on
> > Travis as passing with pulp-smash. Several points of feedback
> > have been addressed.
> >
> > If you're interested in commenting on these PRs or trying them
> > out, please do. I hope to merge after the other taking system
> > maintainers @dalley and @daviddavis have finished their
> > testing/review and barring any other calls for delay or blocking
> > concerns.
> >
> > If there are any questions, issues, or concerns, please reach
> > out, and we can talk through them.
> >
> >
> >
> > On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse
> > <bbouters at redhat.com <mailto:bbouters at redhat.com>> wrote:
> >
> > I put together a prototype and posted the PRs. I'm still
> > working to get Travis happy, but locally 100% of smash tests
> > using these branches. It's worked very reliably for me so
> > far. There are no gaps in the pulp feature set on top of RQ.
> >
> > I hope people test it out and give some feedback. See the
> > commit messages for details on what was done. Here are the
> PRs:
> >
> > https://github.com/pulp/pulp/pull/3454
> > <https://github.com/pulp/pulp/pull/3454>
> > https://github.com/pulp/pulp_file/pull/72
> > <https://github.com/pulp/pulp_file/pull/72>
> > https://github.com/pulp/devel/pull/146
> > <https://github.com/pulp/devel/pull/146>
> > https://github.com/PulpQE/pulp-smash/pull/960
> > <https://github.com/PulpQE/pulp-smash/pull/960>
> >
> > Feel free to send questions here or to the PR. Any feedback
> > is welcome.
> >
> > -Brian
> >
> >
> >
> >
> > On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik
> > <mkovacik at redhat.com <mailto:mkovacik at redhat.com>> wrote:
> >
> > +1 I like RQ and I like
> > http://python-rq.org/docs/testing/
> > <http://python-rq.org/docs/testing/> esp.
> > there's Fakeredis ;)
> >
> >
> > --
> > milan
> >
> >
> > On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse
> > <bbouters at redhat.com <mailto:bbouters at redhat.com>>
> wrote:
> > > Thanks for all the discussion both on list and on irc.
> > After more
> > > investigation, it sounds like there are no feature
> > gaps, but we will need to
> > > incorporate this workaround to cancel a task that is
> > already running.
> > >
> > > The feedback I've heard on the idea is that it's
> > valuable and looks
> > > feasible, but we won't really know until we prototype
> > it a bit. Based on the
> > > technical outline in the previous email, I believe it
> > can be prototyped in a
> > > day or two. I plan to do this soon, once I contribute
> > to a few other
> > > required-for-beta planning items first. I'll post my
> > PR to see what other
> > > think of the change, probably next week.
> > >
> > >
> > >
> > > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley
> > <dalley at redhat.com <mailto:dalley at redhat.com>> wrote:
> > >>
> > >> I meant in the sense that, what is the aftermath when
> > it comes back
> > >> online, and is it screwed up in ways that cause side
> > effects.
> > >>
> > >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet
> > <jaudet at redhat.com <mailto:jaudet at redhat.com>> wrote:
> > >>>
> > >>> > RQ does not support revoking tasks. If you send
> > the worker a SIGINT,
> > >>> > it will finish the task and then stop processing
> > new ones. If you send the
> > >>> > worker SIGKILL, it will stop immediately, but I
> > don't think it gracefully
> > >>> > handles this circumstance.
> > >>>
> > >>> Nothing handles SIGKILL gracefully. Processes can't
> > catch that signal.
> > >>> `kill -9 $pid` sends SIGKILL.
> > >>>
> > >>> If one is looking for a way to gracefully,
> > immediately kill an RQ
> > >>> worker, then SIGTERM may do the trick. Anecdotally,
> > many processes
> > >>> handle this signal in a hurried fashion.
> > Semantically, this is
> > >>> appropriate: SIGINT is the "terminal interrupt"
> > signal (Ctrl+c sends
> > >>> SIGINT), whereas SIGTERM is the "termination signal."
> > >>
> > >>
> > >
> > >
> > > _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> > >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> >
> >
> >
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> >
>
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180508/822d06de/attachment.htm>
More information about the Pulp-dev
mailing list