<div dir="ltr"><div>Hi Bin Li,</div><div><br></div><div>When you perform the failover to the passive standby Pulp what does the status API show for its workers before, during, and after failover? Note the workers and webservers all need to be using the same Redis because the task data flows through Redis.</div><div><br></div><div>I'm wondering if maybe your passive Pulp after failover is indeed receiving the workers that belong to it and that all "initially active" workers are declared missing/dead. Then if the right workers are being shown they all need to be using the same Redis instance (the webservers and the workers).</div><div><br></div><div>Also the "stalling" task should show the "worker" it was assigned to. Sharing that info would help to know where the work was being routed to also.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 18, 2020 at 9:28 AM Dennis Kliban <<a href="mailto:dkliban@redhat.com">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The redis cluster support needs to be added to rq actually[0,1]. Looks like there is an open PR but it hasn't moved forward in a long time[2].<br></div><div><br></div><div>[0] <a href="https://github.com/rq/rq/issues/862" target="_blank">https://github.com/rq/rq/issues/862</a></div><div>[1] <a href="https://github.com/rq/rq/issues/1048" target="_blank">https://github.com/rq/rq/issues/1048</a></div><div>[2] <a href="https://github.com/rq/rq/pull/942" target="_blank">https://github.com/rq/rq/pull/942</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 18, 2020 at 9:07 AM Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>How many instances of Redis are involved? Is every pulpcore-api instance and pulpcore-worker instance pointing to the same redis instance? This is necessary for the work to be routed correctly. </div><div><br></div><div>Pulpcore currently uses redis-py, which does not support connecting to a Redis Cluster[0]. However, we should investigate if it's viable to switch to using redis-py-cluster[1]. <br></div><div><br></div><div>[0] <a href="https://github.com/andymccurdy/redis-py/issues/931" target="_blank">https://github.com/andymccurdy/redis-py/issues/931</a></div><div>[1] <a href="https://github.com/Grokzen/redis-py-cluster" target="_blank">https://github.com/Grokzen/redis-py-cluster</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 13, 2020, 6:28 PM Bin Li (BLOOMBERG/ 120 PARK) <<a href="mailto:bli111@bloomberg.net" target="_blank">bli111@bloomberg.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="white-space:pre-wrap;font-size:small;font-family:"Courier New",Courier,"BB.FixedWidth""><div>Hi Brian,</div>I did a quick test on a active passive pulp 3.1 setup. Two pulp servers are pointing to the same external postgres database. Only one server is active at any time. Redis queue resides on the localhost. The /var/lib/pulp are synced from primary to the contingency host. <div>After I shutdown primary host, I was able to bring up the contingency pulp server and created a repo. Deleting any repo stuck in a waiting state. Then I started primary host and shutdown contingency host, I was able to delete repos I created on the contingency host but all previous delete job continually stuck in the waiting state.</div><div>I am wonder if anything I could do to make this work on contingency host or this setup is not going to work?</div><div><br></div><div>Thanks</div><div><br><div style="font-size:small;font-family:"Courier New",Courier,"BB.FixedWidth""><br><div><div>From: <a href="mailto:pulp-list@redhat.com" rel="noreferrer" target="_blank">pulp-list@redhat.com</a> At: 01/03/20 12:01:44</div>To: <a href="mailto:pulp-list@redhat.com" rel="noreferrer" target="_blank"> pulp-list@redhat.com</a><br>Subject: Pulp-list Digest, Vol 122, Issue 1</div><br><blockquote>Send Pulp-list mailing list submissions to<br> <a href="mailto:pulp-list@redhat.com" rel="noreferrer" target="_blank">pulp-list@redhat.com</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a><br>or, via email, send a message with subject or body 'help' to<br> <a href="mailto:pulp-list-request@redhat.com" rel="noreferrer" target="_blank">pulp-list-request@redhat.com</a><br><br>You can reach the person managing the list at<br> <a href="mailto:pulp-list-owner@redhat.com" rel="noreferrer" target="_blank">pulp-list-owner@redhat.com</a><br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Pulp-list digest..."<br><br><br>Today's Topics:<br><br> 1. Re: pulp3 High availability and disaster recovery (Brian Bouterse)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 2 Jan 2020 16:10:29 -0500<br>From: Brian Bouterse <<a href="mailto:bmbouter@redhat.com" rel="noreferrer" target="_blank">bmbouter@redhat.com</a>><br>To: JASON STELZER <<a href="mailto:jasonstelzer@boomi.com" rel="noreferrer" target="_blank">jasonstelzer@boomi.com</a>><br>Cc: pulp-list <<a href="mailto:pulp-list@redhat.com" rel="noreferrer" target="_blank">pulp-list@redhat.com</a>><br>Subject: Re: [Pulp-list] pulp3 High availability and disaster recovery<br>Message-ID:<br> <<a href="mailto:CAAcvrTGDYCJxcO3TR50Wub1j2Suc6g9Q1_yqjVdsYS_t44qDYw@mail.gmail.com" rel="noreferrer" target="_blank">CAAcvrTGDYCJxcO3TR50Wub1j2Suc6g9Q1_yqjVdsYS_t44qDYw@mail.gmail.com</a>><br>Content-Type: text/plain; charset="utf-8"<br><br>Sorry for the late reply. Each component of Pulp itself can be deployed in<br>HA configurations. Of the services Pulp's processes depend on, Redis is the<br>one service that can't run as a full cluster because RQ doesn't support<br>that yet, so the best you can do is a hot-spare Redis that auto-fails over.<br>That isn't graceful failover so when traffic routes to your hot-spare Redis<br>it has to data and doesn't have the tasking system's data. Those Pulp tasks<br>would be cancelled, and Pulp would be immediately ready to accept new tasks<br>so they could be resubmitted, e.g. Katello resubmits some job failures I<br>believe.<br><br>More docs about this are here:<br><a href="https://docs.pulpproject.org/components.html#architecture-and-deploying" rel="noreferrer" target="_blank">https://docs.pulpproject.org/components.html#architecture-and-deploying</a><br>More questions are welcome; sorry for the slow response. If you can see any<br>way to improve the docs and want to get involved, PRs are welcome!<br><br>-Brian<br><br><br>On Mon, Nov 18, 2019 at 7:37 AM JASON STELZER <<a href="mailto:jasonstelzer@boomi.com" rel="noreferrer" target="_blank">jasonstelzer@boomi.com</a>><br>wrote:<br><br>> For what it is worth, at heart pulp3 is a django app. So, following the<br>> advice for HA and django apps generally works. A lot of it is driven by the<br>> particulars of your use case.<br>><br>> My use case is a little different than yours I'm sure. But in terms of HA<br>> for now I'm good with a balancer and nodes in multiple azs, an RDS db with<br>> failover, and regular db backups.<br>><br>> In my case, the pulp3 server is far enough behind the scenes that even if<br>> there were to be a several hour outage, the impact would be minimal. YMMV.<br>><br>> Others can chime in with pulp3 specifics.<br>><br>> On Fri, Nov 15, 2019 at 11:41 AM Bin Li (BLOOMBERG/ 120 PARK) <<br>> <a href="mailto:bli111@bloomberg.net" rel="noreferrer" target="_blank">bli111@bloomberg.net</a>> wrote:<br>><br>>> Does pulp3 support active/active or active/passive configuration? What is<br>>> the strategy to restore the pulp3 service on a different server if the<br>>> primary is down? Do we have any documentation on this topic?<br>>><br>>> Thanks<br>>> _______________________________________________<br>>> Pulp-list mailing list<br>>> <a href="mailto:Pulp-list@redhat.com" rel="noreferrer" target="_blank">Pulp-list@redhat.com</a><br>>> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a><br>><br>><br>><br>> --<br>> J.<br>> _______________________________________________<br>> Pulp-list mailing list<br>> <a href="mailto:Pulp-list@redhat.com" rel="noreferrer" target="_blank">Pulp-list@redhat.com</a><br>> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <br><<a href="https://www.redhat.com/archives/pulp-list/attachments/20200102/4cc40982/attachment.html" rel="noreferrer" target="_blank">https://www.redhat.com/archives/pulp-list/attachments/20200102/4cc40982/atta<br>chment.html</a>><br><br>------------------------------<br><br>_______________________________________________<br>Pulp-list mailing list<br><a href="mailto:Pulp-list@redhat.com" rel="noreferrer" target="_blank">Pulp-list@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a><br><br>End of Pulp-list Digest, Vol 122, Issue 1<br>*****************************************<br><br></blockquote><br></div></div></div></div>_______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" rel="noreferrer" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div> </blockquote></div> _______________________________________________<br> Pulp-list mailing list<br> <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div>