Weird ethernet port behavior

[Matty Sarro]

Hey everyone. I'm having a weird issue right now with a network port and I
don't know what to make of it.
We are using on an automated provisioning system to deploy RHEL5 on to an HP
DL360 G6 server. The server has two ethernet ports (broadcom netextreme2),
as well as an additional 2 port NIC (intel e1000e).

Eth0 and eth2 are bonded using the bonding module. This physically equates
to the two left most ports. These both work fine, and the bonding drive is
working like gangbusters.

Eth1 is configured to connect to a secondary network. Eth3 is disabled
(onboot=no in /etc/sysconfig/network-scripts/ifcfg-eth3).

Here are the contents of the ifcfg-eth1 and ifcfg-eth3 files:
eth1
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
NETWORK=10.253.163.224
NETMASK=255.255.255.224
IPADDR=10.253.163.242
USERCTL=no

eth3
DEVICE=eth3
USERCTL=no
ONBOOT=no
BOOTPROTO=dhcp

After the bonded pair, we have two ethernet ports left. When we attempt to
plug in what SHOULD be eth1 and restart network services, it gets no
connectivity. After rebooting, when you run ethtool eth1 it shows that eth1
has no link. Running ethtool on eth3 shows that it DOES have link.

Now if we move the cable to the other open port and reboot, running ethtool
on eth1 again shows no link, but link on eth3.

What's even weirder are that the eth1 and eth3 ports are on different
network cards - 1 on the netextreme, the other on the e1000e.

Its like the eth3 designation seems to be following the cable, even though
we have the port disabled. This is the last hitch in the process and I'm
racking my brain trying to figure it out. Any help would be very much
appreciated.

-Matthew