[rhn-users] Net interface flap, Dell 2650, RH ES 3.0 2.4.21-15.ELsmp, Broadcom BCM5701

[Jed Donnelley]

I've been running RH ES 3.0 on a Dell 2650 server with for some time with a 
GigE interface
with the Broadcom NetXtreme BCM5701 technology the tg3 driver, most 
recently with the
2.4.21-9.0.1.ELsmp kernel.  Last week I tried upgrading to the 
2.4.21-15.ELsmp kernel.
When I ran with that newer kernel the network interface on the box went 
into a loop of
going up and down every few seconds.

Here's what I believe is the relevant part of the message log:

May 20 15:58:30 tsi kernel: Linux version 2.4.21-15.ELsmp 
(bhcompile at bugs.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat 
Linux 3.2.3-34)) #1 SMP Thu Apr 22 00:18:24 EDT 2004
...
May 20 15:58:31 tsi kernel: Processors: 4
May 20 15:58:31 tsi kernel: xAPIC support is present
May 20 15:58:31 tsi kernel: Enabling APIC mode: Flat.^IUsing 3 I/O APICs
May 20 15:58:31 tsi kernel: Kernel command line: ro root=/dev/md0
May 20 15:58:31 tsi kernel: Initializing CPU#0
May 20 15:58:31 tsi kernel: Detected 1993.566 MHz processor.
<actually 2 hyperthreaded procesors>
...
May 20 15:58:38 tsi kernel: eth0: Tigon3 [partno(BCM95701A10) rev 0105 
PHY(5701)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:f8:0f:04
May 20 15:58:38 tsi kernel: eth0: HostTXDS[0] RXcsums[1] LinkChgREG[1] 
MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
May 20 15:58:38 tsi kernel: eth1: Tigon3 [partno(BCM95701A10) rev 0105 
PHY(5701)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:f8:0f:05
May 20 15:58:38 tsi kernel: eth1: HostTXDS[0] RXcsums[1] LinkChgREG[1] 
MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
May 20 15:58:38 tsi kernel: ip_tables: (C) 2000-2002 Netfilter core team
May 20 15:58:38 tsi kernel: ip_conntrack version 2.1 (8192 buckets, 65536 
max) - 304 bytes per conntrack
May 20 15:58:38 tsi kernel: tg3.c:v3.1 (April 3, 2004)
May 20 15:58:38 tsi kernel: eth0: Tigon3 [partno(BCM95701A10) rev 0105 
PHY(5701)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:f8:0f:04
May 20 15:58:38 tsi kernel: eth0: HostTXDS[0] RXcsums[1] LinkChgREG[1] 
MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
May 20 15:58:38 tsi kernel: eth1: Tigon3 [partno(BCM95701A10) rev 0105 
PHY(5701)] (PCIX:133MHz:64-bit) 10/100/1000BaseT Ethernet 00:06:5b:f8:0f:05
May 20 15:58:38 tsi kernel: eth1: HostTXDS[0] RXcsums[1] LinkChgREG[1] 
MIirq[1] ASF[0] Split[0] WireSpeed[1] TSOcap[0]
May 20 15:58:38 tsi kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
May 20 15:58:38 tsi kernel: tg3: eth0: Flow control is on for TX and on for RX.
May 20 15:58:38 tsi kernel: tg3: eth0: Link is down.
May 20 15:58:38 tsi kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
May 20 15:58:38 tsi kernel: tg3: eth0: Flow control is on for TX and on for RX.
...
May 20 15:58:43 tsi kernel: tg3: eth0: Link is down.
...
May 20 15:58:46 tsi kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
May 20 15:58:46 tsi kernel: tg3: eth0: Flow control is on for TX and on for RX.
...
May 20 15:58:58 tsi kernel: tg3: eth0: Link is down.
May 20 15:59:01 tsi kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
May 20 15:59:01 tsi kernel: tg3: eth0: Flow control is on for TX and on for RX.
May 20 15:59:02 tsi kernel: tg3: eth0: Link is down.
May 20 15:59:05 tsi kernel: tg3: eth0: Link is up at 1000 Mbps, full duplex.
May 20 15:59:05 tsi kernel: tg3: eth0: Flow control is on for TX and on for RX.
etc.

Here's the output from ethtool for the interface with the problem:

Settings for eth0:
         Supported ports: [ MII ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised auto-negotiation: Yes
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
         Link detected: no

With the older 2.4.21-9.0.1.ELsmp kernel I get the same output except
that I get:
...
         Link detected: yes

with no flapping and with the interface up and functioning.

Has anybody else seen a comparable problem with 2.4.21-15.EL or 2.4.21-15.WS
with a GigE interface with the Broadcom NetXtreme BCM5701 using the tg3 driver?

I wonder if this problem is related to:

http://www.redhat.com/archives/fedora-test-list/2004-April/msg01880.html

?  I'm open to any suggestions.  I certainly don't want a network interface
problem blocking my kernel updates.

--Jed http://www.nersc.gov/~jed/