[rhelv6-list] routing/interface question

[Peter Ruprecht]

John Haxby wrote:
> 
> 
> On 14 January 2011 18:01, Peter Ruprecht <ruprech at jilau1.colorado.edu 
> <mailto:ruprech at jilau1.colorado.edu>> wrote:
> 
>     Hi everyone,
> 
>     I think I'm seeing a difference in behavior between RHEL 5 and 6 on
>     how packets get routed between different subnets on different
>     network interfaces.  Say I have a dual-homed host, with each
>     interface connected to a different physical class C subnet.  The
>     routing table looks like:
> 
>     # netstat -rn
>     Kernel IP routing table
>     Destination     Gateway         Genmask         Flags   MSS Window
>      irtt Iface
>     128.138.140.0   0.0.0.0         255.255.255.0   U         0 0      
>        0 eth1
>     128.138.107.0   0.0.0.0         255.255.255.0   U         0 0      
>        0 eth0
>     169.254.0.0     0.0.0.0         255.255.0.0     U         0 0      
>        0 eth0
>     169.254.0.0     0.0.0.0         255.255.0.0     U         0 0      
>        0 eth1
>     0.0.0.0         128.138.107.1   0.0.0.0         UG        0 0      
>        0 eth0
> 
>     In RHEL5, if I ping the host's 128.138.140.X address from a machine
>     on the 128.138.107. subnet, I can use tcpdump to see the icmp
>     request coming in on eth1, and the reply going out on eth0.  The
>     host is not doing forwarding; that is, there's a 0 in
>     /proc/sys/net/ipv4/ip_forward.
> 
> 
> As someone else pointed out (npanderson at ups.com <http://ups.com>) , 
> this is to do with reverse path filtering.   More specifically, it's to 
> do with a change to rp_filter that happened in RHEL6.  (I tracked down 
> the specific commit a little while ago, I can do it again if you're 
> interested).
> 
> It's instructive to compare npanderson's quote of the RHEL5 docs with RHEL6:
> 
> -----
> rp_filter - INTEGER
>         0 - No source validation.
>         1 - Strict mode as defined in RFC3704 Strict Reverse Path
>             Each incoming packet is tested against the FIB and if the 
> interface
>             is not the best reverse path the packet check will fail.
>             By default failed packets are discarded.
>         2 - Loose mode as defined in RFC3704 Loose Reverse Path
>             Each incoming packet's source address is also tested against 
> the FIB
>             and if the source address is not reachable via any interface
>             the packet check will fail.
> 
>         Current recommended practice in RFC3704 is to enable strict mode
>         to prevent IP spoofing from DDos attacks. If using asymmetric 
> routing
>         or other complicated routing, then loose mode is recommended.
> 
>         The max value from conf/{all,interface}/rp_filter is used
>         when doing source validation on the {interface}.
> 
>         Default value is 0. Note that some distributions enable it
>         in startup scripts.
> -----
> 
> Red Hat are (correctly) setting rp_filter to 1, strict mode.  In this 
> case a packet coming in eth0 will have its source address routed out on 
> the same interface that it came in on (because that's the default 
> route).  However, a packet coming in on eth1 will have it source address 
> routed out on a different interface to the one it came in on and it will 
> be discarded.  Silently.
> 
> This is basically asymmetric routing and is quite possibly not what you 
> want anyway (it messes up TCP flow control) so there are two ways to fix 
> this: stick with asymmetric routing and permit it or fix the asymmetric 
> routing.
> 
> The first one is easiest: in /etc/sysctl.conf change rp_filter=1 to 
> rp_filter=2).  You'll need to load that and restart the network.  It's 
> probably easiest to reboot :-) to be sure.  I suspect that it was not 
> restarting enough things that prevented this change from working before.
> 
> The second one may be simple as simple as adding those routes that 
> should go out on eth1 to the routing table or running some routing 
> daemon.  It depends on your network topology, basically.  This would be 
> the preferred solution if it's practicable.
> 

Thanks for the very detailed and clear explanation, and also to Nathan 
Anderson for originally pointing me at reverse path filtering.  After 
setting rp_filter to 2 (rather than 1 or 0) for the relevant interfaces, 
the routing on the new RHEL 6 system is working in the same way as on 
RHEL 5.  I greatly appreciate your help!

  -Peter