IBM x346 broadcom debacle.
Jimmy Stewpot
squid at oranged.to
Fri Feb 24 10:48:50 UTC 2006
Hello,
I have been working to diagnose some issues with the IBM x346 servers
when using the on board broadcom network cards. The issues occur when I
use both the bcm5700 driver provided by IBM and the tg3 driver provided
by Redhat, Broadcom or the vanilla kernel.
The issue is that when we enable our ICMP Polling service that sends out
several thousand packets over a period of 5minutes to several thousand
different hosts we see that the packets leave the system. When we go and
have a look at the Cisco Catalyst 6509 and do an rspan of the port we
see the ICMP packets returning to the switch port and on the server we
never see those packets with tcpdump on the system.
After some research I found that there are many other people
experiencing similar problems and after my previous post I have been
sent many people asking if I had a resolution. In one way I have a
resolution but it has been compounded further with this issue.
The resolution for the problem was to add an additional Intel
EtherExpress 100 network card into the system and make that the primary
interface. Since doing that and having BOTH broadcom network interfaces
disabled the problem goes away. No more lag, no issues at all.
One thing that I did on the side was update all of the firmware on the
server to the latest BIOS, network firmware and others. Since doing that
the problem was slightly reduced but it was by no means a complete
resolution to the problem.
Now that I am using an intel eepro 100 card in the server it is working
great. However we also have a Veritas netbackup client running on that
server which backs up the data across the second network interface. That
second network interface is again the broadcom on board. This time it is
the second on board interface rather than the first. When the interfaces
are both disabled the problem is non existant. However when we enable
the eth2 interface on board for the netbackup the system starts to lag
even though the routes are all via the eth0 (eepro100) interface.
From seeing those symptoms I am wondering if there could be some sort
of memory interrupt/dma issue on the system that Linux is not reporting
correctly. I am interested to know if anyone else has had similar issues
in the past. Could it also be that the card is running 64bit rather than
in 32bit like the eepro100 card?
Here is a copy of my lspci -v output for any further hardware
investigation. Any ideas or concepts would be great to hear about so
that I can hopefully try and resolve these outstanding issues once and
for all.
===SNIP====
lspci -v
00:00.0 Host bridge: Intel Corp. Server Memory Controller Hub (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: bus master, fast devsel, latency 0
Memory at <ignored> (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] #09 [4105]
00:00.1 Class ff00: Intel Corp. Memory Controller Hub Error Reporting
Register (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: fast devsel
00:02.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
A0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=04, sec-latency=0
I/O behind bridge: 00004000-00004fff
Memory behind bridge: ce000000-cfffffff
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit-
Queue=0/1 Enable-
Capabilities: [64] #10 [0041]
00:04.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
Memory behind bridge: cc000000-cdffffff
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit-
Queue=0/1 Enable-
Capabilities: [64] #10 [0041]
00:05.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
B1 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
Memory behind bridge: ca000000-cbffffff
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit-
Queue=0/1 Enable-
Capabilities: [64] #10 [0041]
00:06.0 PCI bridge: Intel Corp. Memory Controller Hub PCI Express Port
C0 (rev 0a) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
I/O behind bridge: 00005000-0000ffff
Memory behind bridge: c8000000-c9ffffff
Capabilities: [50] Power Management version 2
Capabilities: [58] Message Signalled Interrupts: 64bit-
Queue=0/1 Enable-
Capabilities: [64] #10 [0041]
00:08.0 System peripheral: Intel Corp. Memory Controller Hub Extended
Configuration Registers (rev 0a)
Subsystem: IBM: Unknown device 02dd
Flags: fast devsel
00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
(rev 02) (prog-if 00 [UHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 16
I/O ports at 2200 [size=32]
00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
(rev 02) (prog-if 00 [UHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 19
I/O ports at 2600 [size=32]
00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI
Controller (rev 02) (prog-if 20 [EHCI])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at f0000000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Capabilities: [58] #0a [20a0]
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI
Bridge (rev c2) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
I/O behind bridge: 00003000-00003fff
Memory behind bridge: de000000-dfffffff
Prefetchable memory behind bridge: d0000000-ddffffff
00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
Flags: bus master, medium devsel, latency 0
00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100
Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
Subsystem: IBM: Unknown device 02dd
Flags: bus master, medium devsel, latency 0, IRQ 17
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at <ignored>
I/O ports at 0480 [size=16]
Memory at d8000000 (32-bit, non-prefetchable) [size=1K]
00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
Subsystem: IBM: Unknown device 02dd
Flags: medium devsel, IRQ 17
I/O ports at 0440 [size=32]
01:06.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY
[Radeon 7000/VE] (prog-if 00 [VGA])
Subsystem: IBM: Unknown device 02c8
Flags: bus master, stepping, medium devsel, latency 64, IRQ 20
Memory at d0000000 (32-bit, prefetchable) [size=128M]
I/O ports at 3000 [size=256]
Memory at dfff0000 (32-bit, non-prefetchable) [size=64K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [50] Power Management version 2
02:00.0 PCI bridge: Intel Corp. PCI Bridge Hub A (rev 09) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=02, secondary=03, subordinate=03, sec-latency=48
I/O behind bridge: 00004000-00004fff
Memory behind bridge: ce000000-cfffffff
Capabilities: [44] #10 [0071]
Capabilities: [5c] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [6c] Power Management version 2
Capabilities: [d8] PCI-X non-bridge device.
02:00.2 PCI bridge: Intel Corp. PCI Bridge Hub B (rev 09) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=02, secondary=04, subordinate=04, sec-latency=64
Capabilities: [44] #10 [0071]
Capabilities: [5c] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [6c] Power Management version 2
Capabilities: [d8] PCI-X non-bridge device.
03:03.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100]
(rev 0c)
Subsystem: IBM 10/100 Ethernet Server Adapter
Flags: bus master, medium devsel, latency 64, IRQ 72
Memory at cffff000 (32-bit, non-prefetchable) [size=4K]
I/O ports at 4000 [size=64]
Memory at cffc0000 (32-bit, non-prefetchable) [size=128K]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: [dc] Power Management version 2
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
Subsystem: IBM: Unknown device 02c6
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at cdff0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable-
Capabilities: [d0] #10 [0001]
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 01)
Subsystem: IBM: Unknown device 02c6
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at cbff0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable-
Capabilities: [d0] #10 [0001]
07:00.0 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
Memory behind bridge: c8000000-c9ffffff
Capabilities: [44] #10 [0071]
Capabilities: [5c] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [6c] Power Management version 2
Capabilities: [d8] PCI-X non-bridge device.
07:00.2 PCI bridge: Intel Corp. 80332 [Dobson] I/O processor (rev 07)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=07, secondary=09, subordinate=09, sec-latency=64
I/O behind bridge: 00005000-0000ffff
Capabilities: [44] #10 [0071]
Capabilities: [5c] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Capabilities: [6c] Power Management version 2
Capabilities: [d8] PCI-X non-bridge device.
08:0e.0 RAID bus controller: Adaptec ServeRAID Controller (rev 07)
Subsystem: IBM: Unknown device 028e
Flags: bus master, stepping, 66Mhz, medium devsel, latency 64,
IRQ 38
Memory at c9fff000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=512K]
Capabilities: [c0] Power Management version 2
Capabilities: [d0] Message Signalled Interrupts: 64bit+
Queue=0/1 Enable-
Capabilities: [e0] PCI-X non-bridge device.
===SNIP===
Regards,
Jimmy Stewpot.
More information about the redhat-list
mailing list