[Pulp-list] pulp3 High availability and disaster recovery

[Dennis Kliban]

How many instances of Redis are involved? Is every pulpcore-api instance
and pulpcore-worker instance pointing to the same redis instance? This is
necessary for the work to be routed correctly.

Pulpcore currently uses redis-py, which does not support connecting to a
Redis Cluster[0]. However, we should investigate if it's viable to switch
to using redis-py-cluster[1].

[0] https://github.com/andymccurdy/redis-py/issues/931
[1] https://github.com/Grokzen/redis-py-cluster

On Thu, Feb 13, 2020, 6:28 PM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:

> Hi Brian,
> I did a quick test on a active passive pulp 3.1 setup. Two pulp servers
> are pointing to the same external postgres database. Only one server is
> active at any time. Redis queue resides on the localhost. The /var/lib/pulp
> are synced from primary to the contingency host.
> After I shutdown primary host, I was able to bring up the contingency pulp
> server and created a repo. Deleting any repo stuck in a waiting state. Then
> I started primary host and shutdown contingency host, I was able to delete
> repos I created on the contingency host but all previous delete job
> continually stuck in the waiting state.
> I am wonder if anything I could do to make this work on contingency host
> or this setup is not going to work?
>
> Thanks
>
>
> From: pulp-list at redhat.com At: 01/03/20 12:01:44
> To: pulp-list at redhat.com
> Subject: Pulp-list Digest, Vol 122, Issue 1
>
> Send Pulp-list mailing list submissions to
> pulp-list at redhat.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://www.redhat.com/mailman/listinfo/pulp-list
> or, via email, send a message with subject or body 'help' to
> pulp-list-request at redhat.com
>
> You can reach the person managing the list at
> pulp-list-owner at redhat.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pulp-list digest..."
>
>
> Today's Topics:
>
> 1. Re: pulp3 High availability and disaster recovery (Brian Bouterse)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 2 Jan 2020 16:10:29 -0500
> From: Brian Bouterse <bmbouter at redhat.com>
> To: JASON STELZER <jasonstelzer at boomi.com>
> Cc: pulp-list <pulp-list at redhat.com>
> Subject: Re: [Pulp-list] pulp3 High availability and disaster recovery
> Message-ID:
> <CAAcvrTGDYCJxcO3TR50Wub1j2Suc6g9Q1_yqjVdsYS_t44qDYw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Sorry for the late reply. Each component of Pulp itself can be deployed in
> HA configurations. Of the services Pulp's processes depend on, Redis is the
> one service that can't run as a full cluster because RQ doesn't support
> that yet, so the best you can do is a hot-spare Redis that auto-fails over.
> That isn't graceful failover so when traffic routes to your hot-spare Redis
> it has to data and doesn't have the tasking system's data. Those Pulp tasks
> would be cancelled, and Pulp would be immediately ready to accept new tasks
> so they could be resubmitted, e.g. Katello resubmits some job failures I
> believe.
>
> More docs about this are here:
> https://docs.pulpproject.org/components.html#architecture-and-deploying
> More questions are welcome; sorry for the slow response. If you can see any
> way to improve the docs and want to get involved, PRs are welcome!
>
> -Brian
>
>
> On Mon, Nov 18, 2019 at 7:37 AM JASON STELZER <jasonstelzer at boomi.com>
> wrote:
>
> > For what it is worth, at heart pulp3 is a django app. So, following the
> > advice for HA and django apps generally works. A lot of it is driven by
> the
> > particulars of your use case.
> >
> > My use case is a little different than yours I'm sure. But in terms of HA
> > for now I'm good with a balancer and nodes in multiple azs, an RDS db
> with
> > failover, and regular db backups.
> >
> > In my case, the pulp3 server is far enough behind the scenes that even if
> > there were to be a several hour outage, the impact would be minimal.
> YMMV.
> >
> > Others can chime in with pulp3 specifics.
> >
> > On Fri, Nov 15, 2019 at 11:41 AM Bin Li (BLOOMBERG/ 120 PARK) <
> > bli111 at bloomberg.net> wrote:
> >
> >> Does pulp3 support active/active or active/passive configuration? What
> is
> >> the strategy to restore the pulp3 service on a different server if the
> >> primary is down? Do we have any documentation on this topic?
> >>
> >> Thanks
> >> _______________________________________________
> >> Pulp-list mailing list
> >> Pulp-list at redhat.com
> >> https://www.redhat.com/mailman/listinfo/pulp-list
> >
> >
> >
> > --
> > J.
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <
> https://www.redhat.com/archives/pulp-list/attachments/20200102/4cc40982/atta
> chment.html>
>
> ------------------------------
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
> End of Pulp-list Digest, Vol 122, Issue 1
> *****************************************
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200218/63527a4c/attachment.htm>