OT-hijacked. . .Re: [K12OSN] Solving the bandwidth bottleneck

[Jim McQuillan]

Hmm,

Let me try to explain WHY connecting 2 nics on the same machine to the
same switch is a BAD IDEA.

I'm talking about the case when you are NOT using VLANS.
VLANs effectively split a physical switch into separate switches.

It helps to have an understanding of how packets get from one machine to
another on the same lan.

When 2 computers on a LAN talk to each other, they don't address each
other by IP address.  Deep down, they really address each other with MAC
addresses.

The way a client gets the MAC address of the server is it sends out
an ARP request.  This is an Ethernet broadcast.  It is to see 'Who has'
a specific IP address.

ALL nodes on the network will receive that broadcast.

The server will receive the broadcast on both nics.  AND, since the
server has the IP address in question, it will send out the ARP reply on
both of it's interfaces.  That's because it is the protocol
stack in the kernel that is handling this, not the NIC.

Now, because the server sent out 2 replies, one for each nic, the switch
will see these ARP replies, and add both to its arp table. Depending on
how the arp table code is setup in the switch, it might keep both, or
one might step on the other.

Then, when you try talking to the server, the packet will get to the
switch, and then the switch will decide which port to send it out, based
on its arp tables.  It might send the packet out the correct port, or it
might send the packet out the wrong port, OR, it might send it out both
ports.  Don't forget, we use switches and not hubs, because switches
setup virtual circuits between 2 nodes on the net.  It's the arp table
that allows the switch to know how to setup that circuit.

Sometimes you'll get lucky, and it will just seem to work fine.
Other times, you'll have un-explained packet loss or collisions.

This has NOTHING to do with your choice if IP Subnetting.  This happens
below the IP layer.

So, my advice:  NEVER connect 2 nics on the same server to the same
switch.  Even if it seems to work fine when you do it.  Trouble lurks,
and it won't appear when you are sitting idle with nothing else to do.
The trouble will happen on a friday afternoon, just as you are heading
out to begin a vacation, and the boss calls, because his PC on his desk
can't get to the internet, so he can't check his stock portfolio.

Jim McQuillan
jam at Ltsp.org




On Tue, 10 May 2005, Eric Harrison wrote:

> Jim Kronebusch wrote:
> > > I have two NICs in a server, one on a public IP and one on a
> > > private one
> > > and they are both connected to the same switch. . . is this really a
> > > problem?
> > >
> > > Don't seem to be from here. . .
> >
> >
> > As far as know this definitely breaks more than a couple networking
> > rules :-)
> >
> > If it is working I hate to say change it as the general rule is "if it
> > ain't broke don't fix it".  But you may want to fix that.
> >
> > You should never have 2 networks propogating the same switch.  This
> > should cause major slowdowns and collisions, if not more.
> >
> >
>
> Having two NICs on the same logical network is certainly a no-no.
>
> But in Doug's case above, he specifies that the two NICs are on
> different logical networks (one public, one private). I have the same
> setup on several of my servers.
>
> -Eric
>