From adrianvv at synaq.com Wed Dec 21 11:00:27 2011 From: adrianvv at synaq.com (Adrian van Vuuren) Date: Wed, 21 Dec 2011 13:00:27 +0200 (SAST) Subject: Not all interfaces come up In-Reply-To: <278d85de-7cc7-4e17-a9bb-164bf03ecd25@cms-ah-zcs3.synaq.com> References: Message-ID: <3d17540a-1a39-484c-86cb-0293c032a397@cms-ah-zcs3.synaq.com> Hi, We have a direct lvs setup with 15 or so virtual balancers over 5 IPs with a master and backup pulse (heartbeat). ipvsadm-1.25-9.el6.x86_64 piranha-0.8.5-7.el6.x86_64 What we're finding happens is that one or two (it varies) of the alias interfaces don't come up, seemingly randomly. This happens on a start, a restart, relocating to the backup pulse box, relocating back to the master pulse, etc. Following which the alias interfaces that don't come up need to be brought up manually before those balancers work. Has anyone seen something like this before? Logs don't show any failures or warnings, just making sockets available and creating monitors. Our balancers all follow a similar configuration template for example: virtual balancer-http { active = 1 address = XX:XX:XX:01 eth0:5 vip_nmask = 255.255.255.192 port = 80 use_regex = 0 load_monitor = none scheduler = wlc protocol = tcp timeout = 120 reentry = 30 quiesce_server = 0 server cluster-box1 { address = XX:XX:XX:02 active = 1 weight = 1 } server cluster-box2 { address = XX:XX:XX:03 active = 1 weight = 1 } server cluster-box3 { address = XX:XX:XX:04 active = 0 weight = 1 } } From infantes_car at gva.es Wed Dec 21 11:24:08 2011 From: infantes_car at gva.es (Carlos Infantes) Date: Wed, 21 Dec 2011 12:24:08 +0100 Subject: Not all interfaces come up In-Reply-To: <3d17540a-1a39-484c-86cb-0293c032a397@cms-ah-zcs3.synaq.com> References: <3d17540a-1a39-484c-86cb-0293c032a397@cms-ah-zcs3.synaq.com> Message-ID: <4EF1C1D8.4020706@gva.es> We had the same issue here, and after a lot of work we think that the problem is with the LAN driver. Pulse try to get up the interfaces more fast than the driver can handle and some IPs dont get up. Our "solution" was install RedHat but over XenServer, the hypervisor handles the interfaces without problems. On 21/12/11 12:00, Adrian van Vuuren wrote: > > Hi, > > We have a direct lvs setup with 15 or so virtual balancers over 5 IPs with a master and backup pulse (heartbeat). > ipvsadm-1.25-9.el6.x86_64 > piranha-0.8.5-7.el6.x86_64 > > What we're finding happens is that one or two (it varies) of the alias interfaces don't come up, seemingly randomly. This happens on a start, a restart, relocating to the backup pulse box, relocating back to the master pulse, etc. Following which the alias interfaces that don't come up need to be brought up manually before those balancers work. > > Has anyone seen something like this before? > Logs don't show any failures or warnings, just making sockets available and creating monitors. > > Our balancers all follow a similar configuration template for example: > > virtual balancer-http { > active = 1 > address = XX:XX:XX:01 eth0:5 > vip_nmask = 255.255.255.192 > port = 80 > use_regex = 0 > load_monitor = none > scheduler = wlc > protocol = tcp > timeout = 120 > reentry = 30 > quiesce_server = 0 > server cluster-box1 { > address = XX:XX:XX:02 > active = 1 > weight = 1 > } > server cluster-box2 { > address = XX:XX:XX:03 > active = 1 > weight = 1 > } > server cluster-box3 { > address = XX:XX:XX:04 > active = 0 > weight = 1 > } > } > > > _______________________________________________ > Piranha-list mailing list > Piranha-list at redhat.com > https://www.redhat.com/mailman/listinfo/piranha-list > > -- Carlos Infantes Oanes Unitat d'Actuaci? en Salut P?blica Direcci? General de Salut P?blica Conselleria de Sanitat - GVA Tel.: 961 925 850 From adrianvv at synaq.com Wed Dec 21 11:47:28 2011 From: adrianvv at synaq.com (Adrian van Vuuren) Date: Wed, 21 Dec 2011 13:47:28 +0200 (SAST) Subject: Not all interfaces come up In-Reply-To: <4EF1C1D8.4020706@gva.es> References: Message-ID: <04771cb2-e3d1-46db-9ba0-8a6d12ae7f37@cms-ah-zcs3.synaq.com> Hmmm, wouldn't be the first problem we've had with the card. It didn't handle fragmentation well and (after much investigation) found we needed to disable gro with ethtool. ----- Original Message ----- From: "Carlos Infantes" To: piranha-list at redhat.com Cc: adrianvv at synaq.com Sent: Wednesday, 21 December, 2011 1:24:08 PM Subject: Re: Not all interfaces come up We had the same issue here, and after a lot of work we think that the problem is with the LAN driver. Pulse try to get up the interfaces more fast than the driver can handle and some IPs dont get up. Our "solution" was install RedHat but over XenServer, the hypervisor handles the interfaces without problems. On 21/12/11 12:00, Adrian van Vuuren wrote: > > Hi, > > We have a direct lvs setup with 15 or so virtual balancers over 5 IPs with a master and backup pulse (heartbeat). > ipvsadm-1.25-9.el6.x86_64 > piranha-0.8.5-7.el6.x86_64 > > What we're finding happens is that one or two (it varies) of the alias interfaces don't come up, seemingly randomly. This happens on a start, a restart, relocating to the backup pulse box, relocating back to the master pulse, etc. Following which the alias interfaces that don't come up need to be brought up manually before those balancers work. > > Has anyone seen something like this before? > Logs don't show any failures or warnings, just making sockets available and creating monitors. > > Our balancers all follow a similar configuration template for example: > > virtual balancer-http { > active = 1 > address = XX:XX:XX:01 eth0:5 > vip_nmask = 255.255.255.192 > port = 80 > use_regex = 0 > load_monitor = none > scheduler = wlc > protocol = tcp > timeout = 120 > reentry = 30 > quiesce_server = 0 > server cluster-box1 { > address = XX:XX:XX:02 > active = 1 > weight = 1 > } > server cluster-box2 { > address = XX:XX:XX:03 > active = 1 > weight = 1 > } > server cluster-box3 { > address = XX:XX:XX:04 > active = 0 > weight = 1 > } > } > > > _______________________________________________ > Piranha-list mailing list > Piranha-list at redhat.com > https://www.redhat.com/mailman/listinfo/piranha-list > > -- Carlos Infantes Oanes Unitat d'Actuaci? en Salut P?blica Direcci? General de Salut P?blica Conselleria de Sanitat - GVA Tel.: 961 925 850 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Picture-0 Type: image/png Size: 4728 bytes Desc: not available URL: From rrajaram at redhat.com Wed Dec 21 12:10:01 2011 From: rrajaram at redhat.com (Ranjith Rajaram) Date: Wed, 21 Dec 2011 17:40:01 +0530 Subject: Not all interfaces come up In-Reply-To: <3d17540a-1a39-484c-86cb-0293c032a397@cms-ah-zcs3.synaq.com> References: <3d17540a-1a39-484c-86cb-0293c032a397@cms-ah-zcs3.synaq.com> Message-ID: <4EF1CC99.6030309@redhat.com> Hello, On 12/21/2011 04:30 PM, Adrian van Vuuren wrote: > Has anyone seen something like this before? Can you access the following Red Hat Kbase article https://access.redhat.com/kb/docs/DOC-56762 [Need RH login] Set the following sysctl variables "net.ipv4.conf.all.promote_secondaries = 1" and see if you can reproduce the issue Best regards -- Ranjith Rajaram RHCA, RHCDS Team Lead,Technical Account Manager, Global Support Services. */"Developing Software is humane but supporting them is divine"/*