<div dir="ltr"><div></div><div>It sounds like you may be experiencing issue <a href="https://pulp.plan.io/issues/3135" target="_blank">https://pulp.plan.io/issues/3135</a> </div>From our conversation on IRC, I learned that the hypervisor is acting up and the VMs pause from time to time. So even though the system is not under heavy load it still behaves as though it is. As a result the INactive resource managers think that the active resource manager has become inactive and start being active. What I am still not clear on is why more than 1 resource manager is able to become active at a time. If this is actually happening, then this is a new bug. You could avoid this problem by only running 2 resource managers. Though it would be good to find a reliable way to reproduce this problem and file a bug. </div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Jan 10, 2018 at 6:37 AM, Sebastian Sonne <<a href="mailto:sebastian.sonne@noris.de" target="_blank">sebastian.sonne@noris.de</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello everyone. I have two pulp clusters, each containing three nodes, all systems are up to date (pulp 2.14.3). However, the cluster behavior differs greatly. Let's call the working cluster the external one, and the broken one internal. The setup: Everything is virtualized. Both clusters are distributed over two datacenters, but they're on different ESX-clusters. All nodes are allowed to migrate between hypervisors. On the external cluster, "celery status" gives me one resource manager, on the external cluster I get either two or three resource managers. As far as I understand, I can run the resource manager on all nodes, but should only see one in celery, because the other two nodes are going into standby. Running "ps fauxwww |grep resource_manage[r]" on the external cluster gives me four processes in the whole cluster. The currently active resource manager has two processes, the other ones have one process each. However, on the internal cluster I get six processes, two on each node. >From my understanding, the external cluster works correctly, as the active resource manager has one process to communicate with celery, and one to do work, with the other two nodes only having one active process to communicate with celery and become active in case the currently active resource manager goes down. Oddly enough, celery seems to also disconnect it's own workers: "Jan 10 08:52:36 pulp02 pulp[101629]: celery.worker.consumer:INFO: missed heartbeat from reserved_resource_worker-1@pulp02". As such, I think we can eliminate the network" I'm completely stumped and don't even have a real clue of what logs I could provide, or where to start looking into things. Grateful for any help, Sebastian Sebastian Sonne Systems & Applications (OSA) noris network AG Thomas-Mann-Strasse 16−20 90471 Nürnberg Deutschland Tel <a href="tel:%2B49%20911%209352%201184" value="+4991193521184">+49 911 9352 1184</a> Fax <a href="tel:%2B49%20911%209352%20100" value="+499119352100">+49 911 9352 100</a> <a href="mailto:sebastian.sonne@noris.de">sebastian.sonne@noris.de</a> <a href="https://www.noris.de" rel="noreferrer" target="_blank">https://www.noris.de</a> - Mehr Leistung als Standard Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Jürgen Städing Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689 _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a> </blockquote></div> </div>