[Pulp-dev] Port Pulp3 to use RQ
Brian Bouterse
bbouters at redhat.com
Tue May 8 16:01:13 UTC 2018
On Tue, May 8, 2018 at 11:53 AM, Bryan Kearney <bkearney at redhat.com> wrote:
> ok.. so it is quiesce the old system and then normal updates?
>
Yes, exactly. :)
>
> -- bk
>
> On 05/08/2018 07:27 AM, Brian Bouterse wrote:
> > Here is a look at what the Pulp2 -> Pulp3 necessary things would be
> > w.r.t this change. These could also be automated.
> >
> > 1. Empty the Pulp system of all of it's tasks and stop all Pulp services.
> > 2. Uninstall RabbitMQ or Qpid if its only purpose was to serve the Pulp
> > tasking system. (Satellite uses Qpid in other ways so it would likely
> > keep it in the architecture for other purposes).
> > 3. Uninstall Celery/kombu/billiard/py-amqp (the whole celery stack
> > effectively)
> > 4. upgrade the bits to Pulp3. This will also bring RQ with it
> automatically.
> > 5. Install Redis as a new service in your infra
> > 6. Replace the systemd files for the pulp_workers and
> > pulp_resource_manager. This causes systemd to start RQ instead of Celery.
> > 7. [optional] Configure Redis auth/ssl and configure Pulp's settings
> > file to match if that is part of your environment.
> >
> > Questions/ideas/concerns are welcome.
> >
> >
> >
> > On Tue, May 8, 2018 at 8:48 AM, Bryan Kearney <bkearney at redhat.com
> > <mailto:bkearney at redhat.com>> wrote:
> >
> > what does this look like for upgrading from Pulp2 to Pulp3?
> >
> > -- bk
> >
> > On 05/08/2018 05:34 AM, David Davis wrote:
> > > +1. Thank you @bmbouter and @dalley for working on this.
> > >
> > >
> > > David
> > >
> > > On Mon, May 7, 2018 at 5:37 PM, Daniel Alley <dalley at redhat.com
> <mailto:dalley at redhat.com>
> > > <mailto:dalley at redhat.com <mailto:dalley at redhat.com>>> wrote:
> > >
> > > I've finished my review and resolved all of the 'blocker'
> issues
> > > that were uncovered during testing. Overall, I'm highly
> confident
> > > that this is a better path forwards than the continued use of
> Celery
> > > / Kombu. There are a couple of outstanding edge cases to be
> > > resolved eventually, which I plan to file as issues
> post-merge, but
> > > nothing serious or intractable.
> > >
> > > If there are no objections, I think it would be reasonable to
> merge
> > > this code after this week's beta builds are published (after,
> in
> > > order to avoid major changes during Summit / PyCon prep time).
> > >
> > > Thank you, Brian, for doing the planning and work needed to
> make
> > > this happen. It was a lot of effort and is very highly
> appreciated.
> > >
> > > On Mon, Apr 30, 2018 at 8:28 AM, Brian Bouterse <
> bbouters at redhat.com <mailto:bbouters at redhat.com>
> > > <mailto:bbouters at redhat.com <mailto:bbouters at redhat.com>>>
> wrote:
> > >
> > > Through several rebases, now all PRs are showing the RQ
> PRs on
> > > Travis as passing with pulp-smash. Several points of
> feedback
> > > have been addressed.
> > >
> > > If you're interested in commenting on these PRs or trying
> them
> > > out, please do. I hope to merge after the other taking
> system
> > > maintainers @dalley and @daviddavis have finished their
> > > testing/review and barring any other calls for delay or
> blocking
> > > concerns.
> > >
> > > If there are any questions, issues, or concerns, please
> reach
> > > out, and we can talk through them.
> > >
> > >
> > >
> > > On Fri, Apr 20, 2018 at 4:18 PM, Brian Bouterse
> > > <bbouters at redhat.com <mailto:bbouters at redhat.com>
> > <mailto:bbouters at redhat.com <mailto:bbouters at redhat.com>>> wrote:
> > >
> > > I put together a prototype and posted the PRs. I'm
> still
> > > working to get Travis happy, but locally 100% of smash
> tests
> > > using these branches. It's worked very reliably for me
> so
> > > far. There are no gaps in the pulp feature set on top
> of RQ.
> > >
> > > I hope people test it out and give some feedback. See
> the
> > > commit messages for details on what was done. Here are
> the PRs:
> > >
> > > https://github.com/pulp/pulp/pull/3454
> > <https://github.com/pulp/pulp/pull/3454>
> > > <https://github.com/pulp/pulp/pull/3454
> > <https://github.com/pulp/pulp/pull/3454>>
> > > https://github.com/pulp/pulp_file/pull/72
> > <https://github.com/pulp/pulp_file/pull/72>
> > > <https://github.com/pulp/pulp_file/pull/72
> > <https://github.com/pulp/pulp_file/pull/72>>
> > > https://github.com/pulp/devel/pull/146
> > <https://github.com/pulp/devel/pull/146>
> > > <https://github.com/pulp/devel/pull/146
> > <https://github.com/pulp/devel/pull/146>>
> > > https://github.com/PulpQE/pulp-smash/pull/960
> > <https://github.com/PulpQE/pulp-smash/pull/960>
> > > <https://github.com/PulpQE/pulp-smash/pull/960
> > <https://github.com/PulpQE/pulp-smash/pull/960>>
> > >
> > > Feel free to send questions here or to the PR. Any
> feedback
> > > is welcome.
> > >
> > > -Brian
> > >
> > >
> > >
> > >
> > > On Thu, Mar 22, 2018 at 5:28 PM, Milan Kovacik
> > > <mkovacik at redhat.com <mailto:mkovacik at redhat.com>
> > <mailto:mkovacik at redhat.com <mailto:mkovacik at redhat.com>>> wrote:
> > >
> > > +1 I like RQ and I like
> > > http://python-rq.org/docs/testing/ <
> http://python-rq.org/docs/testing/>
> > > <http://python-rq.org/docs/testing/
> > <http://python-rq.org/docs/testing/>> esp.
> > > there's Fakeredis ;)
> > >
> > >
> > > --
> > > milan
> > >
> > >
> > > On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse
> > > <bbouters at redhat.com <mailto:bbouters at redhat.com>
> > <mailto:bbouters at redhat.com <mailto:bbouters at redhat.com>>> wrote:
> > > > Thanks for all the discussion both on list and
> on irc.
> > > After more
> > > > investigation, it sounds like there are no
> feature
> > > gaps, but we will need to
> > > > incorporate this workaround to cancel a task
> that is
> > > already running.
> > > >
> > > > The feedback I've heard on the idea is that it's
> > > valuable and looks
> > > > feasible, but we won't really know until we
> prototype
> > > it a bit. Based on the
> > > > technical outline in the previous email, I
> believe it
> > > can be prototyped in a
> > > > day or two. I plan to do this soon, once I
> contribute
> > > to a few other
> > > > required-for-beta planning items first. I'll
> post my
> > > PR to see what other
> > > > think of the change, probably next week.
> > > >
> > > >
> > > >
> > > > On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley
> > > <dalley at redhat.com <mailto:dalley at redhat.com>
> > <mailto:dalley at redhat.com <mailto:dalley at redhat.com>>> wrote:
> > > >>
> > > >> I meant in the sense that, what is the
> aftermath when
> > > it comes back
> > > >> online, and is it screwed up in ways that cause
> side
> > > effects.
> > > >>
> > > >> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet
> > > <jaudet at redhat.com <mailto:jaudet at redhat.com>
> > <mailto:jaudet at redhat.com <mailto:jaudet at redhat.com>>> wrote:
> > > >>>
> > > >>> > RQ does not support revoking tasks. If you
> send
> > > the worker a SIGINT,
> > > >>> > it will finish the task and then stop
> processing
> > > new ones. If you send the
> > > >>> > worker SIGKILL, it will stop immediately,
> but I
> > > don't think it gracefully
> > > >>> > handles this circumstance.
> > > >>>
> > > >>> Nothing handles SIGKILL gracefully. Processes
> can't
> > > catch that signal.
> > > >>> `kill -9 $pid` sends SIGKILL.
> > > >>>
> > > >>> If one is looking for a way to gracefully,
> > > immediately kill an RQ
> > > >>> worker, then SIGTERM may do the trick.
> Anecdotally,
> > > many processes
> > > >>> handle this signal in a hurried fashion.
> > > Semantically, this is
> > > >>> appropriate: SIGINT is the "terminal interrupt"
> > > signal (Ctrl+c sends
> > > >>> SIGINT), whereas SIGTERM is the "termination
> signal."
> > > >>
> > > >>
> > > >
> > > >
> > > > _______________________________________________
> > > > Pulp-dev mailing list
> > > > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > <mailto:Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>>
> > > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> > > <https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>>
> > > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > <mailto:Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>>
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> > > <https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>>
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> > >
> >
> >
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
> > https://www.redhat.com/mailman/listinfo/pulp-dev
> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20180508/053497d3/attachment.htm>
More information about the Pulp-dev
mailing list