Issues with Performance
Gavin McDonald
gavitron at gmail.com
Mon Feb 13 22:38:27 UTC 2006
I have seen this problem before on a network of IBM x336s, w/ the BCM
interface. Our analysts determined it was the catalyst router. As I
wasn't directly involved in the diagnosis, I can't provide many details
on how it was resolved, except to say that we escalated it to our Cisco
engineer. Apparently, it was a problem with IOS, not with the servers,
as it manifested even worse on the windows boxen. (same IBM h/ware.)
-G
Regards,
Gavin McDonald
========================
EVI Logistic Enterprises
email: me at gavitron.com
phone: (604) 313-3845
> -----Original Message-----
> From: redhat-list-bounces at redhat.com [mailto:redhat-list-
> bounces at redhat.com] On Behalf Of Jimmy Stewpot
> Sent: Monday, February 13, 2006 7:45 AM
> To: redhat-list at redhat.com
> Subject: Issues with Performance
>
> Hello,
>
> I am currently trying to diagnose a particularly strange issue with a
> Redhat Enterprise Linux v3.0 Update 5 system. The particular issue is as
> follows.
>
> I am attempting to run an ICMP monitoring package from the system to
> monitor all of the devices around a network. (circa 10,000). When the
> ICMP monitoring software is enabled the server becomes unusable via SSH.
> By saying unusable I mean the system is very laggy often taking several
> minutes to respond to a command. However if I am on the system console
> the system is still usable and responds with NO delay.
>
> Originally I suspected that the issue was related to system load or
> network congestion. However I started to monitor the switch ports on the
> catalyst 6509 ports with snmp and the utilisation never reaches above
> 1Mbps.
>
> The system is an IBM x336 with dual 2.8Ghz XEONs, 4gb ram and SCSI
> drives it has dual port broad com 10/100/1000 interfaces. If I monitor
> iostat/vmstat or top I see that the system averages around 90% idle. If
> I look at SAR for a long term performance report I see similar
> information.
>
> On the server it directly connects to the 6509 @ 100Mbps Full duplex
> there are no errors on the interface using the bcm5700 driver provided
> by broadcom. Since looking into the issue I initially thought that the
> problem was with the SSH but the problem appears to be more wide spread.
>
> If I do a rspan on the interface that the server is connected to I see
> that the Ethernet frames go both directions however If I do a tcpdump on
> the server I am not seeing the packets with the syntax
>
> 'tcpdump -vvv -n icmp'.
>
> The SSH sessions and the ICMP return packets not being seen I am sure
> are related. In all cases the ports have been forced to 100-FULL Duplex
> and I have tested at several other port link speeds and modes without
> any success.
>
> One thing that is slightly different to many configurations is that the
> gateway is running CatOS, it then routes in hybrid mode and has VRRP
> configured on the device's gateway. However I do not see why that would
> cause any problems as the routing is not flipping and there are no layer
> 2/3 issues that I can detect.
>
> What I am trying to diagnose/determine is why the Linux box would not
> be seeing the packets returning even though we see them on an rspan as
> leaving the switch interface. There are no iptables rules on the server
> at all and from what I can see the sysctl stuff is all setup correctly.
> When I turn on the debug syslog I still get no error messages or
> notifications.
>
> Where should I be looking now and is there any advice that you guys can
> give me?
>
> Regards,
>
> Jimmy.
>
>
>
>
> ===System Information===
>
> /sbin/lspci -v
> 00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a)
> Subsystem: IBM: Unknown device 02dd
> Flags: bus master, fast devsel, latency 0
> Memory at <ignored> (32-bit, non-prefetchable) [size=4K]
> Capabilities: <available only to root>
>
> 00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting
> Register (rev 0a)
> Subsystem: IBM: Unknown device 02dd
> Flags: fast devsel
>
> 00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
> A0 (rev 0a) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=02, subordinate=04, sec-latency=0
> Capabilities: <available only to root>
>
> 00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
> B0 (rev 0a) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
> Memory behind bridge: ce000000-cfffffff
> Capabilities: <available only to root>
>
> 00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
> B1 (rev 0a) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
> Memory behind bridge: cc000000-cdffffff
> Capabilities: <available only to root>
>
> 00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
> C0 (rev 0a) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
> I/O behind bridge: 00004000-0000ffff
> Memory behind bridge: ca000000-cbffffff
> Capabilities: <available only to root>
>
> 00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended
> Configuration Registers (rev 0a)
> Subsystem: IBM: Unknown device 02dd
> Flags: fast devsel
>
> 00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
> (rev 02) (prog-if 00 [UHCI])
> Subsystem: IBM: Unknown device 02dd
> Flags: bus master, medium devsel, latency 0, IRQ 16
> I/O ports at 2200 [size=32]
>
> 00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
> (rev 02) (prog-if 00 [UHCI])
> Subsystem: IBM: Unknown device 02dd
> Flags: bus master, medium devsel, latency 0, IRQ 19
> I/O ports at 2600 [size=32]
>
> 00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI
> Controller (rev 02) (prog-if 20 [EHCI])
> Subsystem: IBM: Unknown device 02dd
> Flags: bus master, medium devsel, latency 0, IRQ 23
> Memory at f0000000 (32-bit, non-prefetchable) [size=1K]
> Capabilities: <available only to root>
>
> 00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI
> Bridge (rev c2) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
> I/O behind bridge: 00003000-00003fff
> Memory behind bridge: de000000-dfffffff
> Prefetchable memory behind bridge: d0000000-ddffffff
>
> 00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev
> 02)
> Flags: bus master, medium devsel, latency 0
>
> 00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100
> Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
> Subsystem: IBM: Unknown device 02dd
> Flags: bus master, medium devsel, latency 0, IRQ 17
> I/O ports at <ignored>
> I/O ports at <ignored>
> I/O ports at <ignored>
> I/O ports at <ignored>
> I/O ports at 0480 [size=16]
> Memory at c8000000 (32-bit, non-prefetchable) [size=1K]
>
> 00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev
> 02)
> Subsystem: IBM: Unknown device 02dd
> Flags: medium devsel, IRQ 17
> I/O ports at 0440 [size=32]
>
> 01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY
> [Radeon 7000/VE] (prog-if 00 [VGA])
> Subsystem: IBM: Unknown device 02c8
> Flags: bus master, stepping, medium devsel, latency 64, IRQ 20
> Memory at d0000000 (32-bit, prefetchable) [size=128M]
> I/O ports at 3000 [size=256]
> Memory at dfff0000 (32-bit, non-prefetchable) [size=64K]
> Expansion ROM at <unassigned> [disabled] [size=128K]
> Capabilities: <available only to root>
>
> 02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
> Capabilities: <available only to root>
>
> 02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=02, secondary=04, subordinate=04, sec-latency=64
> Capabilities: <available only to root>
>
> 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
> Gigabit Ethernet PCI Express (rev 01)
> Subsystem: IBM: Unknown device 02c6
> Flags: bus master, fast devsel, latency 0, IRQ 16
> Memory at cfff0000 (64-bit, non-prefetchable) [size=64K]
> Capabilities: <available only to root>
>
> 06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
> Gigabit Ethernet PCI Express (rev 01)
> Subsystem: IBM: Unknown device 02c6
> Flags: bus master, fast devsel, latency 0, IRQ 16
> Memory at cdff0000 (64-bit, non-prefetchable) [size=64K]
> Capabilities: <available only to root>
>
> 07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
> (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
> Memory behind bridge: ca000000-cbffffff
> Capabilities: <available only to root>
>
> 07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
> (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=07, secondary=09, subordinate=09, sec-latency=64
> I/O behind bridge: 00004000-0000ffff
> Capabilities: <available only to root>
>
> 08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07)
> Subsystem: IBM: Unknown device 028e
> Flags: bus master, stepping, 66Mhz, medium devsel, latency 64,
> IRQ 38
> Memory at cbfff000 (32-bit, non-prefetchable) [size=4K]
> Expansion ROM at <unassigned> [disabled] [size=512K]
> Capabilities: <available only to root>
>
> Module Size Used by Tainted: P
> bcm5700 105380 2
> ide-cd 34016 0 (autoclean)
> cdrom 32608 0 (autoclean) [ide-cd]
> sg 37260 0 (autoclean)
> autofs4 16984 0 (autoclean) (unused)
> audit 90584 2
> basp 62020 0 (unused)
> microcode 6912 0 (autoclean)
> keybdev 2976 0 (unused)
> mousedev 5656 0 (unused)
> hid 22308 0 (unused)
> input 6176 0 [keybdev mousedev hid]
> ehci-hcd 20776 0 (unused)
> usb-uhci 26988 0 (unused)
> usbcore 81088 1 [hid ehci-hcd usb-uhci]
> ext3 89960 4
> jbd 55060 4 [ext3]
> ips 45252 5
> sd_mod 13712 10
> scsi_mod 114344 3 [sg ips sd_mod]
>
> /sbin/mii-tool
> eth0: 100 Mbit, full duplex, link ok
> eth1: 100 Mbit, full duplex, link ok
>
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
More information about the redhat-list
mailing list