Problems getting 2 NICs to work.

[Kevin Kimmell]

Hi.

I've got several HP Proliant servers running mostly Fedora versions of 
redhat. I'm using kernel 2.4.22-1 right now on the ones that are 
problematic.

These MBs on the HPs have dual gigabit ethernet controllers in them. 
/proc/pci reports them correctly as "Broadcom Corporation NetXtreme 
BCM5704 Gigabit Ethernet (rev 2)".

I've got two public class C networks with two separate ISPs. When I 
initialize eth0 to either of my public IP ranges with their respective 
gateways everything works just fine. So now I'm trying to add an IP from 
the other ISP's range so that I've got multiple pathways to these public 
servers.

Whether I add a virtual copy of eth0 with the other ISPs IP/gateway 
combo or if I plug eth1 into the switch and give it one, networking 
stops working. I should mention that both ISPs come in to the same 
router and are in the smae switch plane so all wires from the servers 
are on the same set of switches.

Here's the route table when things are ok:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
204.117.218.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 204.117.218.254 0.0.0.0 UG 0 0 0 eth0

What the hell is that 169.254.0.0 statement? I've searched hi and lo and 
can't figure it out. I'm making a blind guess that it's an IPv6 thing? I 
didn't put it there and removing it doesn't survive a reboot.

Anyhow, at this point all networking is fine. I can ping and trace out 
and vice versa. Now, when I initialize eth1 here's the table:

Destination Gateway Genmask Flags Metric Ref Use Iface
12.168.88.0 * 255.255.255.0 U 0 0 0 eth1
204.117.218.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth1
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 204.117.218.254 0.0.0.0 UG 0 0 0 eth0

As soon as this is activated, networking stops functioning. I can't ping 
in or out. Am I missing some key step? I know that it works becasue I'm 
running a firewall on another machine that's running a 2.4.20 kernel and 
it was autoconfigured and it's functioning properly.

Can anyone give me a clue?

Thanks,
Kevin