Strange network routing problems

[Broekman, Maarten]

Recently we've run into a couple of situations where our routing tables
are mysteriously being altered.

	x.x.x.# represents a unique static route.
	y.y.y.1 represents the proper gateway for the eth3 interface.
	z.z.z.1 represents the proper gateway for the eth0 interface.
	The z.z.z.1 gateway is the default route for the system

Normally, the routing table should look like this:
# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt
Iface
x.x.x.101   y.y.y.1    255.255.255.255 UGH       0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.32   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.96   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.64   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    0.0.0.0         255.255.255.0   U         0 0          0 eth3
x.x.x.0     0.0.0.0         255.255.255.0   U         0 0          0
eth0
x.x.x.0     y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0     y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
0.0.0.0         z.z.z.1     0.0.0.0         UG        0 0          0
eth0

At some point in the middle of doing an up2date on the server, the
routing table becomes:
# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt
Iface
x.x.x.101   y.y.y.1    255.255.255.255 UGH       0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.32   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.96   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.64   y.y.y.1    255.255.255.224 UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    0.0.0.0         255.255.255.0   U         0 0          0 eth3
x.x.x.0     0.0.0.0         255.255.255.0   U         0 0          0
eth0
x.x.x.0     y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0     y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
x.x.x.0    y.y.y.1    255.255.255.0   UG        0 0          0 eth3
0.0.0.0         y.y.y.2    0.0.0.0         UG        0 0          0 eth3
0.0.0.0         y.y.y.3    0.0.0.0         UG        0 0          0 eth3
0.0.0.0         z.z.z.3     0.0.0.0         UG        0 0          0
eth0
0.0.0.0         z.z.z.2     0.0.0.0         UG        0 0          0
eth0
0.0.0.0         z.z.z.1     0.0.0.0         UG        0 0          0
eth0

And as a result, the up2date fails and the networking services need to
be restarted (service network restart).  This results in the routing
table being:
0.0.0.0         z.z.z.1     0.0.0.0         UG        0 0          0
eth0

The additional static routes are added by starting up our "backup-route"
script.

The .2 and .3 gateways are the physical routers, the .1 is a virtual.
In talking with my networking folks, they said the routers are
configured to do IRDP and that sometimes this might happen if the
systems are configured for DHCP.  However, these are _not_ configured to
do DHCP.

Any ideas?  Has anyone seen this before?  I've seen this on both
physical hardware as well as VMware virtual machines.

Thanks,
Maarten Broekman
Fidelity | Investment Management Technology
Enterprise Platform Services
Office: (617) 563-9756
Cell: (617) 590-8005
Email: maarten.broekman at fmr.com