From lhh at redhat.com Fri Apr 15 20:39:58 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 15 Apr 2005 16:39:58 -0400 Subject: RFC: Piranha + Direct Routing HOWTO v0.2 Message-ID: <1113597598.20618.173.camel@ayanami.boston.redhat.com> Hi, We get a lot of questions about Piranha how to make it work with LVS's direct routing mode, so here is a simple HOWTO which should help make it work without any software patches. Big thanks to Mike McLean in helping us get this down on "paper". Let me know if it works for you (or not). -- Lon -------------- next part -------------- Piranha 0.7.7+ Direct Routing Mini-HOWTO v0.2 Scope: This only contains relevant information on how to make direct routing to work with Piranha, it does not explain how to configure Piranha services. Setting up Piranha: (1) Ensure that the following packages are installed on the LVS directors: * piranha * ipvsadm Ensure that the following packages are installed on the LVS real servers: * iptables * arptables_jf (2) Set up and log in to the Piranha web-based GUI. See the following link: http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/cluster-suite/ch-lvs-piranha.html (3) Configure Piranha for Direct Routing. In the "GLOBAL SETTINGS" tab of the Piranha configuration tool, enter the primary server's public IP address in the box provided. The private IP address is not needed/used for Direct Routing configurations. In a direct routing configuration, all real servers as well as the LVS directors share the same virtual IP addresses and should have the same IP route configuration. Click the "Direct Routing" button to enable Direct Routing support on the Piranha LVS director node(s). (4) Configure services + real servers using the Piranha GUI. (5) Set up the each of the real servers using one of the methods below. =========================================================================== Setting up the Real Servers, method #1: Using arptables_jf How it works: Each real server has the virtual IP address(es) configured, so they can directly route the packets. ARP requests for the VIP are ignored entirely by the real servers, and any ARP packets which might otherwise be sent containing the VIPs are mangled to contain the real server's IP instead of the VIPs. Main Advantages: * Ability for applications to bind to each individual VIP/port the real server is servicing. This allows, for instance, multiple instances of Apache to be running bound explicitly to different VIPs on the system. * Performance. Disadvantages: * The VIPs can not be configured to start on boot using standard RHEL system configuration tools. How to make it work: (1) BACK UP YOUR ARPTABLES CONFIGURATION. (2) Configure each real server to ignore ARP requests for each of the virtual IP addresses the Piranha cluster will be servicing. To do this, first create the ARP table entries for each virtual IP address on each real server (the real_ip is the IP the director uses to communicate with the real server; often this is the IP bound to "eth0"): arptables -A IN -d -j DROP arptables -A OUT -d -j mangle --mangle-ip-s This will cause the real servers to ignore all ARP requests for the virtual IP addresses, and change any outbound ARP responses which might otherwise contain the virtual IP so that they contain the real IP of the server instead. The only node in the Piranha cluster which should respond to ARP requests for any of the VIPs is the current active Piranha LVS director node. Once this has been completed on each real server, we can save the ARP table entries for later. Run the following commands on each real server: service arptables_jf save chkconfig --level 2345 arptables_jf on The second command will cause the system to reload the arptables configuration we just made on boot - before the network is started. (3) Configure the virtual IP address on all real servers using 'ifconfig' to create an IP alias: ifconfig eth0:1 192.168.76.24 netmask 255.255.252.0 \ broadcast 192.168.79.255 up Or using the iproute2 utility "ip", for example: ip addr add 192.168.76.24 dev eth0 As noted previously, the virtual IP addresses can not be configured to start on boot using the Red Hat system configuration tools. One way to work around this is to simply place these commands in /etc/rc.d/rc.local. =========================================================================== Setting up the Real Servers, method #2: Use iptables to tell the real servers to handle the packets. How it works: We use an IP tables rule to create a transparent proxy so that a node will service packets sent to the virtual IP address(es), even though the virtual IP address does not exist on the system. Advantages: * Simple to configure. * Avoids the LVS "ARP problem" entirely. Because the virtual IP address(es) only exist on the active LVS director, there _is_ no ARP problem! Disadvantages: * Performance. There is overhead in forwarding/masquerading every packet. * Impossible to reuse ports. For instance, it is not possible to run two separate Apache services bound to port 80, because both must bind to INADDR_ANY instead of the virtual IP addresses. (1) BACK UP YOUR IPTABLES CONFIGURATION. (2) On each real server, run the following for every VIP / port / protocol (TCP, UDP) combination intended to be serviced for that real server: iptables -t nat -A PREROUTING -p -d \ --dport -j REDIRECT This will cause the real servers to process packets destined for the VIP which they are handed. service iptables save chkconfig --level 2345 arptables_jf on The second command will cause the system to reload the arptables configuration we just made on boot - before the network is started. From lhh at redhat.com Fri Apr 15 20:39:58 2005 From: lhh at redhat.com (Lon Hohberger) Date: Fri, 15 Apr 2005 16:39:58 -0400 Subject: RFC: Piranha + Direct Routing HOWTO v0.2 Message-ID: <1113597598.20618.173.camel@ayanami.boston.redhat.com> Hi, We get a lot of questions about Piranha how to make it work with LVS's direct routing mode, so here is a simple HOWTO which should help make it work without any software patches. Big thanks to Mike McLean in helping us get this down on "paper". Let me know if it works for you (or not). -- Lon -------------- next part -------------- Piranha 0.7.7+ Direct Routing Mini-HOWTO v0.2 Scope: This only contains relevant information on how to make direct routing to work with Piranha, it does not explain how to configure Piranha services. Setting up Piranha: (1) Ensure that the following packages are installed on the LVS directors: * piranha * ipvsadm Ensure that the following packages are installed on the LVS real servers: * iptables * arptables_jf (2) Set up and log in to the Piranha web-based GUI. See the following link: http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/cluster-suite/ch-lvs-piranha.html (3) Configure Piranha for Direct Routing. In the "GLOBAL SETTINGS" tab of the Piranha configuration tool, enter the primary server's public IP address in the box provided. The private IP address is not needed/used for Direct Routing configurations. In a direct routing configuration, all real servers as well as the LVS directors share the same virtual IP addresses and should have the same IP route configuration. Click the "Direct Routing" button to enable Direct Routing support on the Piranha LVS director node(s). (4) Configure services + real servers using the Piranha GUI. (5) Set up the each of the real servers using one of the methods below. =========================================================================== Setting up the Real Servers, method #1: Using arptables_jf How it works: Each real server has the virtual IP address(es) configured, so they can directly route the packets. ARP requests for the VIP are ignored entirely by the real servers, and any ARP packets which might otherwise be sent containing the VIPs are mangled to contain the real server's IP instead of the VIPs. Main Advantages: * Ability for applications to bind to each individual VIP/port the real server is servicing. This allows, for instance, multiple instances of Apache to be running bound explicitly to different VIPs on the system. * Performance. Disadvantages: * The VIPs can not be configured to start on boot using standard RHEL system configuration tools. How to make it work: (1) BACK UP YOUR ARPTABLES CONFIGURATION. (2) Configure each real server to ignore ARP requests for each of the virtual IP addresses the Piranha cluster will be servicing. To do this, first create the ARP table entries for each virtual IP address on each real server (the real_ip is the IP the director uses to communicate with the real server; often this is the IP bound to "eth0"): arptables -A IN -d -j DROP arptables -A OUT -d -j mangle --mangle-ip-s This will cause the real servers to ignore all ARP requests for the virtual IP addresses, and change any outbound ARP responses which might otherwise contain the virtual IP so that they contain the real IP of the server instead. The only node in the Piranha cluster which should respond to ARP requests for any of the VIPs is the current active Piranha LVS director node. Once this has been completed on each real server, we can save the ARP table entries for later. Run the following commands on each real server: service arptables_jf save chkconfig --level 2345 arptables_jf on The second command will cause the system to reload the arptables configuration we just made on boot - before the network is started. (3) Configure the virtual IP address on all real servers using 'ifconfig' to create an IP alias: ifconfig eth0:1 192.168.76.24 netmask 255.255.252.0 \ broadcast 192.168.79.255 up Or using the iproute2 utility "ip", for example: ip addr add 192.168.76.24 dev eth0 As noted previously, the virtual IP addresses can not be configured to start on boot using the Red Hat system configuration tools. One way to work around this is to simply place these commands in /etc/rc.d/rc.local. =========================================================================== Setting up the Real Servers, method #2: Use iptables to tell the real servers to handle the packets. How it works: We use an IP tables rule to create a transparent proxy so that a node will service packets sent to the virtual IP address(es), even though the virtual IP address does not exist on the system. Advantages: * Simple to configure. * Avoids the LVS "ARP problem" entirely. Because the virtual IP address(es) only exist on the active LVS director, there _is_ no ARP problem! Disadvantages: * Performance. There is overhead in forwarding/masquerading every packet. * Impossible to reuse ports. For instance, it is not possible to run two separate Apache services bound to port 80, because both must bind to INADDR_ANY instead of the virtual IP addresses. (1) BACK UP YOUR IPTABLES CONFIGURATION. (2) On each real server, run the following for every VIP / port / protocol (TCP, UDP) combination intended to be serviced for that real server: iptables -t nat -A PREROUTING -p -d \ --dport -j REDIRECT This will cause the real servers to process packets destined for the VIP which they are handed. service iptables save chkconfig --level 2345 arptables_jf on The second command will cause the system to reload the arptables configuration we just made on boot - before the network is started. -------------- next part -------------- -- Taroon-list mailing list Taroon-list at redhat.com http://www.redhat.com/mailman/listinfo/taroon-list