From rodrick.r.brown at bofasecurities.com Fri Sep 1 15:13:41 2006 From: rodrick.r.brown at bofasecurities.com (Brown, Rodrick R) Date: Fri, 01 Sep 2006 11:13:41 -0400 Subject: Cleaning up stale handles in /etc/mtab Message-ID: <5F08B160555AC946B5AB743B85FF406D05ABF277@ex2k.bankofamerica.com> Not sure what exactly happened on this server but it looks like the LUNS were pulled w/o removing the volumes correctly or something of that sort. # df df: `/vob/gcib/its': Too many open files in system df: `/vob/lftech/artefacts': Too many open files in system df: `/vob/rmt/mktrisk_portal': Too many open files in system df: `/vob/gcib/gfx_fxpb_client_facing': Too many open files in system df: `/vob/gcib/gfx_fxpb_workflow': Too many open files in system df: `/vob/rmt/credittrend': Too many open files in system df: `/vob/adv/re_reporting': Too many open files in system df: `/vob/datamgmt/atlas_p2': Too many open files in system df: `/vob/gcib/gfx_rad': Too many open files in system ... ... Does anyone know of a method to fix this without rebooting the box thanks? Thanks. --- Rodrick R. Brown UNIX Platform Support (SME) Banc of America Securities LLC. Global Trading Infrastructure (GCIB) 100 West 33rd ST. 3rd Flr. New York, NY 10001 Mail Code: NY1-509-03-18 Office: 646 733 4473 Cell: 646-261-5286 From mgalgoci at redhat.com Fri Sep 1 15:18:18 2006 From: mgalgoci at redhat.com (Matthew Galgoci) Date: Fri, 1 Sep 2006 11:18:18 -0400 Subject: Cleaning up stale handles in /etc/mtab In-Reply-To: <5F08B160555AC946B5AB743B85FF406D05ABF277@ex2k.bankofamerica.com> References: <5F08B160555AC946B5AB743B85FF406D05ABF277@ex2k.bankofamerica.com> Message-ID: > Date: Fri, 01 Sep 2006 11:13:41 -0400 > From: "Brown, Rodrick R" > Reply-To: redhat-sysadmin-list at redhat.com > To: redhat-sysadmin-list at redhat.com > Cc: linux-lvm at redhat.com > Subject: Cleaning up stale handles in /etc/mtab > > Not sure what exactly happened on this server but it looks like the LUNS > were pulled w/o removing the volumes correctly or something of that > sort. If you've got processes in D waiting on those volumes I think you might be hosed beyond recovery without reboot. Having said that, you could try bumping up /proc/sys/fs/file-max to something insanely high and then cat /proc/mounts > /etc/mtab After that, I could only make wild guesses as to what needs to happen next. -- Matthew Galgoci IS Production Operations Red Hat, Inc 919.754.3700 x44155 From greg at netops.gvtc.com Fri Sep 22 21:46:06 2006 From: greg at netops.gvtc.com (Greg Swift) Date: Fri, 22 Sep 2006 16:46:06 -0500 Subject: lvs providing simple web service on top of gfs cluster Message-ID: <4514599E.3040407@netops.gvtc.com> Hi. okay. This is my first try at this, but I'm setting up a rather standard (per the documentation) setup. I'm running on rhel4u4. I want a LVS (2 boxes) router setup ontop of a 4 box active GFS cluster that will provide these services: HTTP(S), FTP, POP, and IMAP (well actually I don't know that i'll configure the IMAP to be available, but meh might as well include it). To start it off all i've configured is the back 4 boxes accessing a GFS share for the web sites, and the apache services for port 80 and 443. I can get on a box on this private network and access all the sites just fine by directing straight to their actual IPs. Next I installed and setup the lvs (piranha based) routers. The steps literally include this: 1: up2date --installall=rhel-i386-es-4-cluster 2: set net.ipv4.ip_forward to 1 in /etc/sysctl.conf and to save a reboot do 'sysctl -w net.ip4.ip_forward=1' 3: piranha-passwd 4: service piranha-gui start 5: configure web interface, (if you want to see it i will show lvs.cf, but i'd rather not push it into a mailing list archive) 6: scp /etc/sysconfig/ha/lvs.cf root at box2:/etc/sysconfig/ha/lvs.cf 7: service pulse start I've configured the firewall to allow the traffic I want through (basically anywhere can access anywhere via port 80), but have tried this with the firewall turned off. A quick check at the cli shows: [root at ament ~]# ipvsadm -L -n IP Virtual Server version 1.2.0 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 216.177.160.9:80 wlc persistent 120 -> 172.16.1.124:80 Masq 1 0 0 -> 172.16.1.123:80 Masq 1 0 0 -> 172.16.1.121:80 Masq 1 0 0 -> 172.16.1.122:80 Masq 1 0 0 although i guess that tells every1 what my lvs.cf looks like... heheh at this point if i try and access 216.177.160.9:80 the InActConn field increments for the 1.124 box, but thats it also, i don't know if its supposed to show up there, but i dont see anything in netstat supporting this. (nothing is listening on port 80, but i realize a port forwarding firewall won't necessarily show that). Here is what happens when you try and access port 80 via telnet: [root at shiva etc]# telnet 216.177.160.9 80 Trying 216.177.160.9... telnet: connect to address 216.177.160.9: No route to host telnet: Unable to connect to remote host: No route to host but [root at shiva etc]# ping 216.177.160.9 PING 216.177.160.9 (216.177.160.9) 56(84) bytes of data. 64 bytes from 216.177.160.9: icmp_seq=0 ttl=64 time=0.254 ms To specify my skill set, I am an rhce, but I realize that i'm probably doing something wrong. I did follow the lvs/piranha documentation on rh's site, and have an open case w/ them. anyone else know any gotchas or have some recommendations about what might be the solution? -greg -- ?While it is possible to change without improving, it is impossible to improve without changing.? -anonymous ?only he who attempts the absurd can achieve the impossible.? -anonymous --- what's this? http://scanner.gvtc.com !DSPAM:45146c6587616309012694! From jbourne at hardrock.org Sat Sep 23 04:19:20 2006 From: jbourne at hardrock.org (James Bourne) Date: Fri, 22 Sep 2006 22:19:20 -0600 (MDT) Subject: lvs providing simple web service on top of gfs cluster In-Reply-To: <4514599E.3040407@netops.gvtc.com> References: <4514599E.3040407@netops.gvtc.com> Message-ID: On Fri, 22 Sep 2006, Greg Swift wrote: > Next I installed and setup the lvs (piranha based) routers. The steps > literally include this: > 1: up2date --installall=rhel-i386-es-4-cluster > 2: set net.ipv4.ip_forward to 1 in /etc/sysctl.conf and to save a reboot do > 'sysctl -w net.ip4.ip_forward=1' > 3: piranha-passwd > 4: service piranha-gui start > 5: configure web interface, (if you want to see it i will show lvs.cf, but > i'd rather not push it into a mailing list archive) > 6: scp /etc/sysconfig/ha/lvs.cf root at box2:/etc/sysconfig/ha/lvs.cf > 7: service pulse start > > I've configured the firewall to allow the traffic I want through (basically > anywhere can access anywhere via port 80), but have tried this with the > firewall turned off. I believe you will have to nat the internal network as well I believe, to mangle the packets going out with the IP of the redirector box (216.177.160.9) before the packages will return, otherwise they may come back from one of your 172.16 IPs which would give you the No route to host error. Regards james > > A quick check at the cli shows: > [root at ament ~]# ipvsadm -L -n > IP Virtual Server version 1.2.0 (size=4096) > Prot LocalAddress:Port Scheduler Flags > -> RemoteAddress:Port Forward Weight ActiveConn InActConn > TCP 216.177.160.9:80 wlc persistent 120 > -> 172.16.1.124:80 Masq 1 0 0 > -> 172.16.1.123:80 Masq 1 0 0 > -> 172.16.1.121:80 Masq 1 0 0 > -> 172.16.1.122:80 Masq 1 0 0 > > although i guess that tells every1 what my lvs.cf looks like... heheh > at this point if i try and access 216.177.160.9:80 the InActConn field > increments for the 1.124 box, but thats it > also, i don't know if its supposed to show up there, but i dont see anything > in netstat supporting this. (nothing is listening on port 80, but i realize a > port forwarding firewall won't necessarily show that). > > Here is what happens when you try and access port 80 via telnet: > > [root at shiva etc]# telnet 216.177.160.9 80 > Trying 216.177.160.9... > telnet: connect to address 216.177.160.9: No route to host > telnet: Unable to connect to remote host: No route to host > > but > > [root at shiva etc]# ping 216.177.160.9 > PING 216.177.160.9 (216.177.160.9) 56(84) bytes of data. > 64 bytes from 216.177.160.9: icmp_seq=0 ttl=64 time=0.254 ms > > > To specify my skill set, I am an rhce, but I realize that i'm probably doing > something wrong. I did follow the lvs/piranha documentation on rh's site, and > have an open case w/ them. anyone else know any gotchas or have some > recommendations about what might be the solution? > > -greg > > -- James Bourne | Email: jbourne at hardrock.org UNIX Systems Administration | WWW: http://www.hardrock.org Custom UNIX Programming | Linux: The choice of a GNU generation ---------------------------------------------------------------------- "All you need's an occasional kick in the philosophy." Frank Herbert From greg at netops.gvtc.com Mon Sep 25 14:16:47 2006 From: greg at netops.gvtc.com (Greg Swift) Date: Mon, 25 Sep 2006 09:16:47 -0500 Subject: lvs providing simple web service on top of gfs cluster In-Reply-To: References: <4514599E.3040407@netops.gvtc.com> Message-ID: <4517E4CF.2030605@netops.gvtc.com> >> I've configured the firewall to allow the traffic I want through >> (basically anywhere can access anywhere via port 80), but have tried >> this with the firewall turned off. > > I believe you will have to nat the internal network as well I believe, to > mangle the packets going out with the IP of the redirector box > (216.177.160.9) before the packages will return, otherwise they may come > back from one of your 172.16 IPs which would give you the No route to > host > error. you mean "iptables -t nat -A POSTROUTING -j MASQUERADE" ? Did that, didn't make a difference in the web service, but the "real" servers can now get out to the rest of the world. -greg -- ?While it is possible to change without improving, it is impossible to improve without changing.? -anonymous ?only he who attempts the absurd can achieve the impossible.? -anonymous --- what's this? http://scanner.gvtc.com !DSPAM:4517f7aa87611848210579!